9
$\begingroup$

A well-known problem in YAML is a type-inference issue in parsing where a string is misinterpreted as a boolean. This is known as the "Norway problem", because it occurs when a field or array entry intended to hold an ISO-3166-2 country code as a string is given the code for Norway — NO.

countries:
- SE
- NO
- FI

YAML accepts yes and no as true and false values, and so when processed the expected string value in the field is instead a boolean. The syntax highlighting above should indicate that.

Norway is the most noted source of these issues, but it would also happen for Ontario, and for values expected to include the strings "yes", "true", "false" (Unix commands), "off" (not leg), "null" (a surname), "nan" (a relative), and some others. A similar, but separate, issue can arise with version numbers, where 1.2.1 and 1.2 are different types, or ratios, where 1:12 is the float 1.2 but 5:60 is a string.

While YAML is a human-editable serialisation format, reminiscent issues could arise in Perl, PHP, shell scripts, and a number of other unequivocal programming languages with unmarked string literals, as well as some other configuration and transfer formats. Later versions of YAML attempted to remove much of this overloading, and it and similar constructions are generally frowned upon — but still present in real-world systems.

There are variations, but versions of this sort of issue have arisen enough that the choices leading to them must have value. What conditions lead to these decisions, in YAML or elsewhere, and when (if ever) is this sort of "smart" value parsing a worthwhile trade-off?

$\endgroup$
4
  • 3
    $\begingroup$ Relavent $\endgroup$
    – mousetail
    Commented May 27, 2023 at 6:28
  • 1
    $\begingroup$ It's not clear to me what you mean by "'off' (not leg)". Are you referring to cricket terminology, perhaps? $\endgroup$ Commented May 21 at 21:52
  • $\begingroup$ Being on one or other side of a line drawn through both middle stumps, yes. $\endgroup$
    – Michael Homer
    Commented May 21 at 22:03
  • $\begingroup$ hitchdev.com/strictyaml/why/implicit-typing-removed $\endgroup$
    – user76284
    Commented Jun 19 at 18:35

3 Answers 3

11
$\begingroup$

It is a somewhat reasonable idea to provide convenient facilities for presumed common cases, but when done carelessly, it tends to backfire in unexpected situations. Javascript is notoriously plagued by this problem. Introducing features like this makes the language generally simpler to use. That's why you can see this in scripting and configuration languages, where it had been intended to facilitate simple tasks. However, the benefits tend to eventually get outweighted by the confusing and inconvenient experiences with unaccounted use-cases. I don't believe there's any further motive behind such features other than the near-sightedness of the language authors.

$\endgroup$
4
  • 5
    $\begingroup$ "It makes easy things easier, and hard things impossible." $\endgroup$ Commented Jul 27, 2023 at 2:54
  • $\begingroup$ @JohnGordon is that a quote you've found elsewhere? I like it! I'd love the source, and if it's you, to quote you :-) $\endgroup$
    – 0atman
    Commented Dec 4, 2023 at 9:15
  • 1
    $\begingroup$ @0atman It's a half-remembered quote about something on Windows -- possibly Visual Basic? It's a deliberate snarky take on an earlier quote something like "the easy things should be easy, and the hard things should be possible". $\endgroup$ Commented Dec 4, 2023 at 13:31
  • $\begingroup$ @JohnGordon brilliant, that's VB in a nut shell! thank you! $\endgroup$
    – 0atman
    Commented Dec 13, 2023 at 16:43
4
$\begingroup$

All programming languages (in a very broad sense, including markup and configuration languages, domain-specific languages, etc) can be seen as a way for humans to express ideas in a way that a computer can act on.

One of the challenges this poses is that human thought is imprecise, fuzzy, and contextual; whereas electronics are (at least by design) precise, logical, and deterministic. This leads to an endless series of design decisions: when do we insist that the programmer must "think like a computer", and when do we try to teach the computer to "think like a human"?

For example, an assembly language might assign mnemonics to locations in memory, but a higher-level language allows the programmer to ignore memory layout completely, and just "declare a variable". To allocate the correct memory for that variable, the compiler needs to know its type, but may provide facilities to infer that type from its surrounding context.

Most of the time, these are powerful features, that help programmers to express their ideas; but sometimes they hide details that are important. The Java manual links to this style guide discussing the pros and cons of using the var keyword introduced in Java 10. One of its examples of the dangers of the feature is where a literal could represent multiple types:

// ORIGINAL
byte flags = 0;
short mask = 0x7fff;
long base = 17;

// DANGEROUS: all infer as int
var flags = 0;
var mask = 0x7fff;
var base = 17;

This is essentially what is happening in the "Norway problem": rather than requiring the user to mark all string values with quote marks (or all non-string values with some other syntax), YAML requires parsers to infer the type of expressions based on specific values. When the values are unambiguous, this makes the syntax look a lot more like what a human would naturally write - a list of strings can be as simple as:

- item one
- item two
- item three

And a boolean setting can be:

spline_reticulation: on

But this syntax is inherently ambiguous. The language specification defines which interpretation should be followed, but it's not always the interpretation that a human would choose with knowledge of the context.

Automatic type coercion can have similar problems. For instance, PHP is designed for use with HTTP, a string-based protocol, so allows users to treat numeric strings directly as numbers: "1" + "2" === 3. But exactly when to apply that coercion is not always clear-cut, e.g. if "0" == 0 and "0.0" == 0 are both true, is "0" == "0.0" also true? Neither answer is going to match intentions in 100% of cases.

There is something of a paradox: the more you try to hide the complexity from the user, the more complexity you have to introduce. Every time we allow the computer to assume the intention of the human, rather than demanding explicit information, we run the risk of it making a wrong assumption.

$\endgroup$
2
$\begingroup$

The design tradeoff obviously has been to sacrifice explicit declaration to save on its notational cost/overhead (declaring types, marking literals) for the benefit of "ease of use" (precisely for users to not have to learn and adhere to additional notational requirements, annotation which is there to enable type inference/checks).

May seem worthwhile to create/accept this problem when it helps more people to write more in the format/language with less of a barrier of entry (main goal being adoption), and may not seem worthwhile if your configurations break and may cause harm/damage/costs for no other reason than ambiguities/collisions/confusions with whatever people happen to want to configure/input. Including the costs to fix/repair/clean-up, find/deploy workarounds, etc.

In this Yaml case, it may not be necessarily the case that there's some type deserialization applied, as code could read the Yaml with all the values as strings, or checking/converting, or wired to know about countries and load all the list items as strings. But in some cases, let's say if it were

country: no

it may remove knowledge about someone's or something's country, it is not reconstructable from the data after the fact what was meant here, and even if it were, no good way to represent it. Is this then the only place for "no", or to apply quotation marks for all country codes, next to false for legitimate cases of missing country code (or should that be null or none which are also not ISO 3166-2 codes but Yaml doesn't know that any string/value in countries is supposed to be a 3166-2 code vs. arbitrary strings, and so on)? Here trying to use notational indicators to identify a type for type checking (to encode meaning). Yaml as a relatively primitive structuring format is not designed to care too much about precise expression of meaning.

$\endgroup$
1
  • 1
    $\begingroup$ YAML may position itself in its official specification as a "structuring format", that leaves type handing to "the application", but in practice, implementations are libraries which automatically infer a wide range of types. Applications which use YAML files for configuration or input absolutely expect the result from those libraries to imply meaning, not just structure. $\endgroup$
    – IMSoP
    Commented Jun 15 at 9:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .