25
$\begingroup$

Alas, when developing my language I may have made some decisions that turned out less than ideal in hindsight. Some I managed to fix in a backwards compatible way, but others will require breaking changes.

I want to avoid a situation like Python 3 though where after 10 years still much of the ecosystem relies on the old version. Other languages have released many major versions without as much drama. What techniques can be used to speed up the transition period?

$\endgroup$
4

5 Answers 5

48
$\begingroup$

Other languages have released many major versions without as much drama. What techniques can be used to speed up the transition period?

We thought about this A LOT on the C# design team. A few thoughts off the top of my head:

(1) Make your language really unpopular, or make the breaking change before the language is popular. If there's hardly any existing code, there's not much to break. If there are hardly any existing users, you can contact them directly and get them to patch their code.

This sounds silly but its true. I often think about how the precedence rules of && and || are messed up in a great many languages descended from C because Ritchie didn't want to do a breaking change for the dozens of users C had at the time, and saddled literally millions of users with the consequences of that decision for decades to follow.

(2) A corollary of (1) that we used to great effect at Facebook: have control of 99.9% of the source code of your language. We introduced MANY breaking changes to Hack when I was on the Hack team. Since 99.9% of the Hack code in the world was in the same repo as the compiler, before we introduced a breaking change we would simply rewrite every piece of code that would have broken.

These work great but what if your language is in the unfortunate position of being enormously successful with millions of users and billions of lines of source code in the world, running key business processes that absolutely must not break?

(3) Don't do it. Bend over backwards to avoid breaking changes.

But sometimes it is inevitable.

(4) Make the breaking changes only affect code that is obscure, unlikely, or almost certainly wrong.

Examples from C#:

  • In C# 1, A(B<C, D>(E)) parses as "call A with two arguments, B<C and D>(E)". In C# 2, it parses as "call A with one argument, a call to generic method B<C, D> with argument E.

    Did anyone actually write that code in C# 1? Probably not. But still, the parser has a bunch of heuristics and lookahead rules that try to mitigate the impacts of these sorts of changes, just in case.

  • We took a breaking change so that "access to modified closure" stopped being a thing for foreach loops.

    Anyone who was broken by that was almost certainly doing it wrong in the first place.

And so on.

(5) Decide ahead of time what breaks are acceptable. Another example from C#, when we added lambdas in C# 3, the overload resolution rule was: if a lambda-to-delegate conversion would cause the body to have an error, the conversion is invalid:

void M(Func<int, int> f) {}
void M(Func<string, int> f) {}
...
M( x => x.Length );

if x is int, x.Length is an error, not so if x is string, so overload resolution chooses the second overload.

But this rule is a factory for new breaking changes! If we ever add a Length property to int then this legal program becomes an illegal program because overload resolution has no basis upon which to prefer one overload and gives an ambiguity error. With this rule about lambdas, ANY change to ANY rule of C# or any new member of any type could potentially turn a working program into an ambiguous one, and we explicitly decided during the design of C# 3 that lambdas were worth that pain.

(6) Warn people ahead of time where breaks will happen, example, since day one the documentation for GetHashCode on System.String has said that Microsoft reserves the right to change the hash code for a given string at any time, for any reason. But in .NET 1, the string hash code never changed, and people started using it as a key not in an in-memory hash table, but in DATABASE hash tables. When a later version of .NET used a different hash algorithm for strings, all those people broke, but they were forewarned if they'd read the documentation.

(7) That was still bad though. To prevent that from happening ever in the future within Microsoft, the hash algorithm was changed so that for debug builds of .NET, every build had a different string hashing algorithm. (The build number was the seed of the hash.) Anyone who took a dependency on a thing that was likely to break got broken within 24 hours, not an entire version or two later.

(8) Pursue a multi-release strategy for breaking changes. In version N, make use of the breaking feature a warning and recommend the replacement. In version N+1, make it an error. In version N+2, remove the original version entirely. Having an [Obsolete] attribute or similar is useful.

There's a lot more to say here but that's all I have time for today; I hope that was helpful. Let me know if you have questions.

$\endgroup$
12
  • 12
    $\begingroup$ The solution you stated in 7 is actually quite good. If you tell people explicitly that something WILL change, but they observe that it doesn't, then they ignore the documentation for some reason, and thinks that's how it is supposed to work. Making it actually change on every release would prevent them from doing something they know is wrong and is actually their own fault. $\endgroup$
    – Nelson
    Jul 14, 2023 at 1:45
  • 2
    $\begingroup$ @Nelson: Bold of you to assume people read the documentation ;) (Which is why, of course, point 7 is an excellent point) $\endgroup$ Jul 14, 2023 at 7:14
  • 10
    $\begingroup$ "I often think about how the precedence rules of && and || are messed up in a great many languages descended from C" – It looks to me like in C, && binds more tightly than ||, which is exactly what I would expect. I do notice, though, that == binds more tightly than the bitwise operators &, |, and ^, which definitely is counterintuitive—an expression like x & 3 == 2 will parse as x & (3 == 2) instead of (x & 3) == 2. Is that perhaps what you were thinking of? $\endgroup$ Jul 14, 2023 at 10:47
  • 1
    $\begingroup$ Interestingly, SQLite adopted C's bitwise operators but changed the precedence to be higher than relationals, allowing typical bitmasking comparisons to use fewer parentheses. $\endgroup$
    – dan04
    Jul 14, 2023 at 16:53
  • 2
    $\begingroup$ @DanielWagner: We did a variation of this at Facebook, yes. We used the compiler's parser to also build a search-and-replace tool that could rapidly find instances of to-be-deprecated patterns in the many-million-LOC repo. $\endgroup$ Jul 15, 2023 at 1:23
20
$\begingroup$

Make breaking changes opt-in ahead of time

This is the route taken by Swift and Python, for example. Python lets you do this at the source level, per-file, with the special form

from __future__ import breaking_change

where breaking_change is the name of an upcoming change that is not yet enabled by default. Swift does this per-module with the compiler flag -enable-upcoming-feature.

Using this approach means that projects can incrementally enable new features as they update the code. It's also possible to alert users of a breaking change without rejecting the code yet: some compiler warnings in Swift 5 warn that "this is an error in Swift 6."

$\endgroup$
4
  • 3
    $\begingroup$ While such approaches are generally a good idea, they can be hard to implement and have very negative consequences on the overall compiler architecture. Also, the analysis itself can be expensive but having it optional is likely a bad idea. $\endgroup$
    – feldentm
    Jul 13, 2023 at 20:04
  • $\begingroup$ Java uses the same commandline flag approach for preview features $\endgroup$
    – Seggan
    Jul 15, 2023 at 0:53
  • $\begingroup$ @feldentm: Some kinds of compilation-mode changes may be awkward to process efficiently, but even many tricky cases could be handled by having code pass through an initial step which is based upon an old parser, and slightly reformats code or adds annotations so as to work with a new one. For example, if an old version of a language specified that applying the "pasting" operator to "+" and "++" would yield "++" and "+" (performing string concatenation and re-tokenizing), a first-phase program could process things in that manner but add annotations to prevent... $\endgroup$
    – supercat
    Jul 24, 2023 at 15:19
  • $\begingroup$ ...things like macro substitutions from being improperly repeated in the main compiler. $\endgroup$
    – supercat
    Jul 24, 2023 at 15:19
11
$\begingroup$

A few more strategies.

Editions

Rust handles breaking changes via an edition mechanism.

First of all, unless otherwise explicitly specified, the edition is 2015 -- the first stable release of the language. Hence, infinite backward compatibility.

Secondly, to avoid the Python2/Python3 debacle, editions are restricted to local changes, mostly syntactic, so that the difference between editions of various libraries is erased by the front-end, and compiled libraries are compatible with each others.

It's a limited approach, but it works really well. Some old libraries are still in edition 2015, when there's been editions 2018 and 2021 already, and recent versions of the compiler can still compile them all in a single application.

Fix-It

If the ecosystem is responsive, automating the transition may be good enough. Go did it a lot in the early days, with go fix automatically "fixing up" your code to transition from old to new version.

It does require a new release of the library code -- hence the need for a responsive ecosystem -- but it takes little effort on each maintainer's thanks to the automation.

$\endgroup$
9
$\begingroup$

Create a version before the breaking changes that is mostly forward compatible with them

Fully updating a large code base takes a long time. However, feature requests and bugs do not go on pause and will also need attention during this time. This will lead to constant duplicate work and merge conflicts for those committed to performing the upgrade.

For companies, the upgrade project will tend not to get enough priority and the work needed to finish the process will only increase over time as the amount of new code grows faster than the amount of upgraded code. For open source projects there is also little motivation for doing duplicate work which also leads to these projects lasting a long time if they ever complete.

The solution is to offer a version of your language that has all the new replacement features but hasn't removed the old ones yet. That way, teams can gradually replace uses of the "old way of doing things" with "the new way" while still being able to pay attention to urgent features and bugs that may appear while they are working. New code can already be written with the new system so the amount of upgrade work will not increase over time.

Many languages follow this system, PHP for example normally have all the new features in the last minor version before the next major version.

This doesn't work for every type of feature, but any small part that can be finished before committing to a major upgrade makes the transition a little easier.

$\endgroup$
2
  • 1
    $\begingroup$ Swift has also done this to an extent: in Swift 5.5 and earlier, the places where you could use existential types were very limited, and there was no special marker. Swift 5.6 allowed you to mark them with any, Swift 5.7 added new situations where existentials are allowed and required the word any in those situations only, and then Swift 6.0 will require any for all existentials. $\endgroup$
    – Bbrk24
    Jul 13, 2023 at 14:19
  • $\begingroup$ I do not see why Java is an example. Besides not having real minor versions, Java 8 seem to be more of a fork than an old version. Actually, Java 8 is a good counter example to your statement as lots of projects are stuck there and need to be replaced entirely. $\endgroup$
    – feldentm
    Jul 13, 2023 at 20:02
4
$\begingroup$

Living on head, parsing immediate past

Or plan to break and support from onset. That is, and to use the Python example, say that any .py file is "current" Python version, but your compilar can accepts and compile older .py1 or .py2 source files, that will have the older functionality somewhat emulated in new semantics.

This will work for major versions, but breaking changes in the level of new keywords or complete new and breaking string type may warrant a major release anyways.

For smaller changes, a directive or pragma may indicate that file in particular is following a specific major-minor release of language.

Something like:

from __past__ import lang.3.4
// or
#pragma langversion 3.4

Your new compiler version can then emit warnings or rejections about that old code. You will need to support only two immediate versions of language, as any programmer can use each successive version of the compiler to upgrade the code.

As a new major or minor version is released, the (now) old code upgrade is released as a separated "lint", or even better, as source-to-source compiler. With these "upgrade tools", your user base will have a very incentive to leave the past in the past.

Have a very good LSP for your language

I have a friend whose programming skills got rusty and asked me to work on a pair programming hobby project to remedy this. For this project, I selected a somewhat old utility that both of us use.

Why is this relevant? Because the project is C# language, originally written half a decade ago. We settleted to using VSCode, and the old code, opened in the new IDE, showed a lot of null warnings and suggestions of how to fix these, and how to use more idiomatic ways to manipulate substrings, records, enumerations...

I'm learning a lot, while fixing and upgrading these things, a warning or tip at time.. I'm literally upgrading the code, and not even thinking in these terms.

There is something when code upgrating is a pleasurable experience.

$\endgroup$
1
  • $\begingroup$ +1 for the 'pleasurable experience' idea! $\endgroup$
    – Pablo H
    Jul 18, 2023 at 12:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .