Questions tagged [parsing]
For questions related to the parsing stage of code compilation or interpretation. Parsing is the process of converting text-based code into the constituent logic.
46
questions
10
votes
1
answer
236
views
How to tolerate syntax errors in a typed language?
In my tokenizers and parsers, once I find a syntax error (like an unexpected token) I usually throw a fatal exception that finishes parsing of the input source.
The downside of how I usually do it is ...
- 769
11
votes
1
answer
195
views
What exactly is Pratt parsing used for and how does it work?
I've come across the term “Pratt parsing”. The only thing I know is that it's an algorithm (or a pattern, a technique) used to parse expressions.
How does it work? I'd like to see the intuition behind ...
- 557
9
votes
3
answers
1k
views
Do parsers typically need access to all tokens?
Do parsers typically operate on the entire array/list of tokens available in memory, or are the tokens often streamed one by one as they are recognized? What influences the decision?
- 557
9
votes
1
answer
627
views
How does Rust parse ambiguous macros?
Rust's macros require them to be delimited by either parentheses, braces, or brackets, such as println!(arg) or foo! { bar, baz }...
- 2,594
2
votes
3
answers
226
views
How do different template languages handle parsing?
In template languages, most of the content is written out as-is, with only certain special markers for expression evaluation or flow control. In my attempts to develop a language of this style, I've ...
- 1,722
6
votes
2
answers
157
views
How to determine the meaning of a polymorphic operator?
Some operators have multiple meanings.
For example:
- represents the prefix negative, or it can represent the infix ...
- 2,552
-2
votes
1
answer
190
views
How to provide a sensible error message for `load (load s0, s1), s2` when parsing assembly? [closed]
Suppose somebody completely misunderstands how assembly language works and, instead of writing:
load s1, s2
load s0, s1
He writes:
...
- 928
-1
votes
1
answer
216
views
Look behind in parsers [closed]
If Sandwich's AST generator finds that the first token is an identifier, one of two cases can be true:
Function call
Variable assignment
It stores the identifier and checks if the next token is a = ...
- 9
17
votes
6
answers
8k
views
Does Python's semicolon statement ending feature have any unique use?
Python does not usually use explicit line-ending characters; the parser can almost always figure out where a statement ends from the positioning of newlines. However, Python's syntax allows you to use ...
- 2,451
16
votes
2
answers
2k
views
How do lexers/parsers distinguish between nested generics and bitshifts?
The lexical grammar of Java has a special case for the the > character. Normally, tokens are formed based on the longest-match rule, so that an input string of <...
- 16.3k
7
votes
1
answer
217
views
Techniques for resilient parsing in the face of mismatched brackets?
Suppose you want to build an IDE-grade parser for a language with highly uniform and nestable syntax (e.g. Lisp). Without a lot of "special" constructs like top-level functions, you can't ...
- 121
5
votes
1
answer
1k
views
How can the shunting yard algorithm be adapted to handle currying?
The shunting yard algorithm is used to parse infix expressions into a syntax tree, or prefix/postfix notation. Wikipedia provides pseudocode for how to implement it, but notes:
This implementation ...
- 8,563
2
votes
1
answer
158
views
What are the exact requirements of not requiring forward declarations everywhere?
Some languages like C and C++ require defining the names before using them, though the definition could sometimes be only a header specifying something is a struct or function without the body. Some ...
- 2,010
2
votes
1
answer
131
views
How do I explain to a syntax highlighter to highlight `a` differently in `load s0, a` (where it is a hexadecimal constant) and `regbank a`? [duplicate]
In the syntax highlighter I have implemented in my PicoBlaze Simulator in JavaScript, the assembly code
address 0
load s0, 0
load s0, a
regbank a
is highlighted as ...
- 928
10
votes
2
answers
244
views
What are the advantages of parser combinators?
Motivation
There seems to be something popular in the rust community called parser combinators.
I temporarily use a traditional recursive descent parser, my syntax is not yet stable, composition means ...
- 2,552
7
votes
7
answers
986
views
Pros and cons of whitespace or comments being included as tokens
What are the pros and cons of including whitespace and comments as tokens in the tokenizer output?
For example:
value= 6+ 7 // A comment
Would result in these ...
- 225
4
votes
1
answer
126
views
Do let bindings need an assignment operator for disambiguation?
Let's say I want to allow (conservative) let bindings, with optional type ascription. Most imperative languages would put a = or ...
user743
4
votes
1
answer
152
views
What sort of "syntax error recovery" is possible for this indentation based language?
I am thinking of how to write the parser/compiler so it works as a VSCode Language Server Extension, so I can have it show syntax errors, do autocomplete, and other things, in VSCode. One thing that I'...
- 655
8
votes
1
answer
362
views
Parsing `do`-`while`, `while`-`do`-`end` and `do`-`end`
Consider the Lua programming language, which has the following constructs involving the keywords do and end:
...
- 288
21
votes
7
answers
6k
views
Why do most languages use the same token for `EndIf`, `EndWhile`, `EndFunction` and `EndStructure`?
In my programming language, AEC, I am using keywords EndIf and EndWhile with the same meaning as in SmallBasic, and I am also ...
- 928
11
votes
5
answers
753
views
What are the pros and cons of a compiler with a single-pass parser?
Assume you are tasked with writing a compiler for a language that can be parsed in a single pass, like C, but does not necessarily have to be. What are the pros and cons for doing this? Would you do ...
- 2,588
12
votes
3
answers
2k
views
What are the pros and cons of single-pass compilers?
Okay, we all know the standard compiler architecture: a batch program with multiple passes for parsing, optimizing, and generating code.
Suppose you wanted to go all Turbo Pascal and implement a ...
- 2,588
4
votes
2
answers
317
views
Does a simple syntax stack based language need a parser?
I have been working on a standard for a simple stack based language for a few weeks now. It has an extremely simple syntax.
...
- 43
4
votes
1
answer
223
views
Is PicoBlaze assembly language a context-sensitive language?
I know one example of context-sensitivity in programming languages is the so-called typedef problem. Basically, the words can be different types, and that influences how a sentence may be parsed. For ...
- 928
5
votes
2
answers
170
views
How can a compiler's parse errors be tested?
There are fairly straightforward ways to test a compiler's behaviour on valid programs: most obviously, build a suite of programs to run through the compiler, and know what should happen when they run....
- 10.9k
2
votes
3
answers
132
views
How can I track source spans during parsing and interpretation?
I want to implement error messages that point out where in your source code a parse error or runtime error is. However, when I convert the token stream to my AST, I lose the information about the ...
- 1,633
11
votes
2
answers
381
views
What are some caveats to be careful about when implementing the `?:` (ternary conditional) operator?
Thus far, I see two things that can go wrong when implementing the ?: (ternary conditional) operator:
PHP famously incorrectly implemented the ...
- 928
5
votes
2
answers
432
views
Can something like a Donkey Sentence exist in a programming language?
I am interested, whether, in some programming language, there can be something like a Donkey Sentence, a sentence that defies straightforward attempts to translate it into a formal language, but is ...
- 928
8
votes
3
answers
130
views
Relative precedence of let and if expressions
In a language with standard let and if expressions, what should be the their precedence such that they can be nested without parentheses?
For example, I want to be able to parse both of these ...
- 249
-5
votes
1
answer
98
views
What kind of a lexer (tokenizer) and parser would be useful for standardizing mailing addresses? [closed]
This is a question about designing and implementing a language of street addresses.
In the input high-level interface language, many syntaxes are allowed.
In the output low-level implementation ...
- 283
7
votes
1
answer
368
views
How are hand-written parsers organised, and how do they work?
Below, we have an example of two extremely simple languages and a relationship between the two languages:
Inputlanguage
(-1, 0)
(0, 1)
(1, 0)
(0, 1)
(0, -1)
(1, 0)
(1, 0)
Outputlanguage
"left&...
- 283
5
votes
2
answers
119
views
What is it that makes parsing shell so hard?
Parsing shell scripts is notoriously challenging. There is a long history of subtle parsing bugs in implementations (e.g.), some of them long-lived, and of static analysis tools that settle for very ...
- 10.9k
4
votes
2
answers
102
views
In a grammar specification, how to exclude characters?
I want to specify in my grammar that an identifier could be any character except for a list of characters (that is have a deny list instead of the most common allow list).
I've seen the common ...
- 199
4
votes
1
answer
91
views
How would I allow a variety of different syntaxes for passing arguments into MathJax or LaTeX commands? [closed]
What is LaTeX
If you already know what LaTeX is, then skip down the the section labeled "sloppy syntax".
There is a type-setting language known as LaTeX.
LaTeX even has its very own Stack ...
- 283
3
votes
1
answer
80
views
Context-free grammar for a language where indentation marks code blocks?
I have the following context-free grammar:
...
- 6,736
10
votes
1
answer
221
views
How to parse a language where functions are full sentences?
I'm designing a language that looks like this:
...
- 6,736
8
votes
1
answer
116
views
When should I use a hand-written parser over a parsing library?
There are plenty of libraries out there that provide tools like LL parsers, combinator parsers, ebnf parsers and more. However, these may not always be suited to the grammar of a programming language.
...
- 1,747
8
votes
1
answer
159
views
How to distinguish custom operators from each other?
How might one go about building a parser that can distinguish custom operators in prefix, infix or postfix positions.
For example:
...
- 157
8
votes
3
answers
151
views
Representing parenthesis in an AST
This language is implemented in Rust using a lot of enums for things like expressions. Some of the planned validation rules involve checking how parenthesis are used -- where they're needed and are ...
- 180
6
votes
1
answer
282
views
How exactly does requiring a semicolon after `EndIf` and `EndWhile` make recovering from parsing errors easier?
In my programming language, AEC, unlike in Ada or VHDL, there is no semicolon after the EndIf token. That is, you can put the semicolon ...
- 928
10
votes
2
answers
411
views
What is the difference between an Abstract Syntax Tree and a Parse Tree?
Compiling and interpreting programming languages typically involves parsing text into a tree that represents the different groupings of syntactic elements. The hierarchy represents which syntactic ...
- 7,285
10
votes
5
answers
204
views
Allowing `}` as a statement terminator in an expression-based language?
My language is expression based; many of its language constructs are not statements, but rather expressions that return a value. However, since everything is an expression, my grammar looks like
...
- 1,633
12
votes
5
answers
189
views
What rules can be used to infer statement endings for a language where they are implicit?
In a lot of older C-family languages, lines of code (statements) must be ended with an explicit character (generally a semicolon). However, newer languages support inferring where statement endings ...
- 2,451
9
votes
3
answers
195
views
How can I handle interpolated expressions in string literals?
I've written what I think is a fairly typical compiler frontend: I have a lexer that produces a list of tokens, then I give that list to a recursive-descent parser that produces the syntax tree. This ...
- 10.9k
20
votes
5
answers
597
views
What are the advantages of requiring forward declaration of methods/fields like C/C++ does?
In C and C++ (and probably other languages), you need to have at least a declaration of functions and fields before you can use them:
...
- 1,722
10
votes
6
answers
510
views
How can ambiguity with unary and binary operators be resolved?
Most languages use +/- for addition and subtraction, in addition to using +/...
- 4,088