I think there is actually a pretty straightforward way to achieve this, but it depends on the compiler being designed to make testing easy. Particularly, errors (and other diagnostics) must be structured data, not just strings, and the compiler needs an API which returns diagnostics in a data structure rather than directly printing them to stderr.
None of these things is particularly difficult to achieve. I'll use the compiler for my own language, Papyri, as an example:
- Diagnostics, including syntax errors, are represented as an enum type (source).
- Diagnostics are collected into a list during compilation (source), instead of being output immediately when they occur.
- Then, test cases for diagnostics can consist of an input as a string, and an enum variant for which diagnostic is expected (source; these examples use a macro to make the tests easier to write).
I'll run through your bullet points to discuss how this strategy accounts for them:
there are many more ways for the code to be wrong than right, but not all of them are interesting;
A test suite for the compiler's diagnostics can be comprehensive if it covers every code path which emits a diagnostic. Having diagnostics represented by an enum makes it easier to ensure full coverage without writing excessively many tests for "uninteresting" cases which are already covered. In practice, many parse errors will only be emitted in one place; for diagnostics reported in multiple places, coverage can be ensured using the IDE's "find all occurrences" tool, or using a code coverage tool.
a test program that's rejected for e.g. a type error instead of the intended parse error is still a wrong result; there can be multiple feasible parse errors arising from one input, but it should reliably produce the "right" one; and in the right place(s);
Each test passes if and only if the expected enum variant is present in the list after compilation; occurrences of other diagnostics don't affect the test result.
In the Papyri compiler, I'm not also testing that the diagnostic is associated with the correct source span, but that's mainly because this would make test-writing more difficult, and the test cases are simple enough that there is only one place that the expected diagnostic could occur.
So if the diagnostic occurs at all then it's very likely in the right place, and I don't feel the need to also test that it does occur in the right place. If you do think it would be worth it, though, then a test could be a triple of (source, expected enum variant, expected span).
a compiler change that causes a different parse error to be produced is likely a problem; but error messages can be expected to change over time to be more useful to the programmer in ways that the semantics of a working program won't, but updating the entire test suite when that happens is fraught;
By using enum variants, the tests are stable with respect to changes in the wording of error messages, while still catching bugs where the wrong kind of diagnostic is emitted. Therefore there is no need to update the test suite when the wording of an error message changes, only when new kinds of diagnostic are introduced, or the conditions on which some kind of diagnostic should be reported are changed.
both rejected and accepted inputs are important.
Accepted inputs (i.e. those without parse errors) will be covered by all the other test cases which test for a particular result instead of a particular diagnostic being emitted. I don't find it that useful to write test cases where the input is valid and the test passes only if no diagnostics are emitted; it's more useful to also require the test to pass if the output matches some expectation.
So yes, it's important to test that parse errors aren't emitted for valid inputs, but this is something you get "for free" when you test that valid inputs have correct compilation results. Therefore I don't think it needs to be part of the strategy for testing diagnostic reporting.
