12
$\begingroup$

In my tokenizers and parsers, once I find a syntax error (like an unexpected token) I usually throw a fatal exception that finishes parsing of the input source.

The downside of how I usually do it is that:

  • I do not get a resulting AST; everything parsed before the error is discarded.
  • in a potential Language Server, I would normally not progress to type checking unless I cache the last type checking after a successful parsing of a program.

I am aware that parsers like jQuery Esprima and luaparse (if I recall correctly) are able to tolerate these fatal syntax errors.

I already use a diagnostics collection per Source, but most syntax errors throw a fatal exception. How can I tolerate such syntax errors (i.e. ignore the error and move to the next parsing)?

$\endgroup$
4
  • 5
    $\begingroup$ I think this question would be more focused if you asked only "should ...", or only "how ...". You could ask both questions separately if you want. $\endgroup$
    – kaya3
    Dec 4, 2023 at 23:17
  • 3
    $\begingroup$ By "tolerate", do you mean "recover from" during parsing (in order to try to decide whether the rest of the code has more errors)? Or are you actually trying to design a language that accepts code that doesn't meet the strict language spec, and tries to make some sense of it? $\endgroup$ Dec 5, 2023 at 7:31
  • $\begingroup$ @KarlKnechtel The former (there are no related error recovery questions apparently) $\endgroup$
    – Hydroper
    Dec 5, 2023 at 11:01
  • 1
    $\begingroup$ The Eclipse Java Compiler (ECJ) also does this. See for example this answer and this wiki. $\endgroup$
    – Lii
    Jan 4 at 7:49

2 Answers 2

9
$\begingroup$

Usually, you would have anchor sets like '}' while within a block and you would only skip tokens until you reach a token in that set. While this might cause more errors, it works reasonably well in practice. Especially, if taken into account during language design.

In later phases like semantic analysis, you need to provide some error objects that you can perform your analyses on. Usually, you wouldn't raise any more errors there or look into error AST nodes.

However, be aware that you cannot always provide a meaningful error type that does not create other issues. Again, in practice common errors are mistyping or imbalanced braces or things like that within a method body. If your language is designed to tolerate such things. Things that are particularly bad are global type inference and other global properties that get affected when removing the content of a method body or initializer or similar.

Finally, if you have a look at how different IDEs work, you will notice that the value they provide is usually reduced a lot while confronted with actual parse errors and it is just somewhat reduced while confronted with semantic analysis errors.

Regarding the language server part, you should consider not updating your state while there are syntactic errors. At least you should try how such an implementation works out in practice before trying more sophisticated approaches.

$\endgroup$
1
$\begingroup$

You can’t parse a program with syntax errors. So what you do is modify the program so there are no syntax errors, and parse that. In a simple C program

int x = 0
int y = 0;

you have a blatant syntax error. A clever compiler will realise that adding a semicolon in the right place fixes it. So it displays an error, and compiles as if that ; was there. If you wrote

Int x = 0;

with capital I a clever compiler will realise that with a lowercase i everything is fine and compile that. In each case an IDE might suggest to fix the problem.

There are cases where a syntax error or other error completely changes the program. Like an extra or missing parenthesis. The compiler can remove or insert one. If programs usually consist of functions, and the language can identify the start of a function, that’s a good point to assume that another function would have ended just before the start of a new one.

So in general, a good compiler will make changes that remove syntax errors, and will extract as much useful information as possible, to be able to continue compiling without undue error messages.

If there is a syntax error with some type, the compiler will often introduce an error type, which can be compiled without error messages. If we have a declaration that x is an int and y’s type contained a syntax error, then x + y can be compiled without an error message. The compiler assumes that errors it displays like the one for y will eventually be fixed and then x + y will be fine.

$\endgroup$
3
  • 2
    $\begingroup$ So how does one actually do that? Assuming that there already is a clever compiler that does all that seems to skip the question. These examples all seem obvious for a human who knows the answer, but how would one approach writing/defining a program/parser/compiler/… to make the same judgement? $\endgroup$ Dec 30, 2023 at 7:23
  • $\begingroup$ @MisterMiyagi Well, here is an article "Resilient LL Parsing Tutorial": matklad.github.io/2023/05/21/resilient-ll-parsing-tutorial.html , by the orignal author of rust-analyzer. $\endgroup$
    – Joker_vD
    Jan 5 at 1:37
  • $\begingroup$ @Joker_vD I'm personally well aware of some strategies, and that there are many more out there. But this answer should contain some to address the question. $\endgroup$ Jan 5 at 6:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .