21
$\begingroup$

Okay, so this might be a silly question. I'm a big fan of functional programming paradigms in higher-level scripting languages, so I'm probably using Rust wrong. However, I'm new to Rust and I'm curious why this isn't a thing. If someone could explain why this isn't a language feature (or provide a solid technical explanation for why it's a bad idea), I'd appreciate it!

So, I'm working my way through the Rust Book. So far, I'm on board with the "evaluating a block as an expression given its last line is an expression business." It seems kind of LISP-y or Swift-y to me. Anyway, Rust lets you do things like this...

let x = {
    let tmp: u32 = 1;
    tmp + 1
};

println!("The value of x is {x}");
// the value is 2

You can also loop over an array, providing a block where you can perform an operation with each element...

let my_arr = [1, 2, 3, 4, 5];

for item in my_arr {
    println!("The value of item is {item}");
}
// the value of item is 1-5

So... given the block expression rules above (coupled with Rust's seeming incredible power to infer strictly typed values), why is it that we can't use for x in y as a way to produce a mapped array of values? I know I'm not that strong with compiled languages, but shouldn't we be able to do something like so?

let my_arr = [1, 2, 3, 4, 5];

let new_arr = for item in my_arr { item + 1 };

for item in new_arr {
    println!("The value of item is {item}");
}
// ???

... which, in my little Rust head-canon here, should result in a new array of values [2, 3, 4, 5, 6]. It's my assumption that, if we can infer the value of variable x in the first example, we should be able to infer the member values of the array new_arr and provision for it, accordingly. Is there a reason why this wouldn't work in Rust? Is it contrary to some design principle of the language that I've yet to discover?

$\endgroup$
7
  • $\begingroup$ The reason it doesn't work is because, to use something in a for loop it specifically needs to implement IntoIter which consumes self. Some methods implement an iter() method that only borrows self and allows you to iterate over it multiple times but there is no trait for that and for loops will never use it unless you call it explicitly. $\endgroup$
    – mousetail
    Aug 31, 2023 at 15:35
  • 3
    $\begingroup$ @mousetail Remember not to answer with comments! $\endgroup$ Aug 31, 2023 at 15:36
  • $\begingroup$ @RydwolfPrograms It doesn't really answer the question, the question is if it could be done in a different way, not why it currently works this way $\endgroup$
    – mousetail
    Aug 31, 2023 at 15:37
  • 1
    $\begingroup$ I don't understand why the two other things (block expressions and for loops) would suggest that the third thing (list comprehensions) should work. The value of a for loop is (), i.e. the unit value, so let new_arr = for ... will declare a variabel new_arr with the value (). $\endgroup$
    – kaya3
    Aug 31, 2023 at 15:48
  • 1
    $\begingroup$ You can use .iter().map(|x|x+1) to perform the effect you want $\endgroup$
    – mousetail
    Aug 31, 2023 at 15:59

4 Answers 4

34
$\begingroup$

Manual memory management is complicated

Rust is a systems programming language. While it is very much possible to use Rust for general-purpose application development, it is primarily specialized to the task of low-level systems programming. The most obvious example of this is its use of manual memory management.

Since Rust aims to achieve full memory safety within its safe subset, much of the language is built around structured mechanisms for controlling memory allocation and lifetimes. In many languages, these concerns are largely handled by the programming language, which makes it both possible and easy to, say, materialize a value from nothing and not worry about where it’s being allocated, who owns it, and how long it lives. The answers to those questions are both simple and uniform: the value is allocated in the heap, the runtime owns its memory, and it lives until it is no longer reachable. Rust does not have such luxuries, and this does not just mean that things are less automatic, it also means they are more nuanced.

In Rust, one can allocate memory in numerous different ways, and values’ lifetimes can be controlled by any of several different mechanisms. There are many different gradations of ownership, each with their own implications and tradeoffs. This is not a shortcoming of the language, it’s the point: people doing systems programming care about the distinctions between the potential options. The purpose of Rust is to give such programmers low-level control over these mechanisms while retaining safety, and in exchange, the programmer must be exquisitely explicit about each of the choices they make.

List comprehensions are not fundamentally incompatible with this attitude, but they are also far from straightforward. Here are a few examples of questions they raise:

  • Where does the memory for the list returned by a comprehension come from? How long does it live for? What happens when it is freed?

  • If the programmer is mapping over an existing structure, is the structure’s memory reused, consuming it in the process? If so, how can the language ensure that memory is not inadvertently accessed from within the body of the loop while the structure is somehow in an invalid state?

  • Since values can escape the body of a list comprehension, how is the memory for local variables within the loop body managed? Can the space used by the loop be linear in the number of iterations? Do values need to be copied at the end of each iteration?

  • In Rust, iteration is overloadable via the Iterator trait, which has 75(!) methods, each of which come with ownership requirements. How can list comprehensions be translated into uses of a comparable API while still being lifetime-correct?

  • Given Rust’s exacting control over memory and lifetimes, no data structure can be one-size-fits-all. How can the type of structure produced by a comprehension be controlled by the user? How can the translation of the comprehension accommodate data structures with different requirements, like fixed sizes or custom allocators?

This is not an exhaustive list, but it should give you some appreciation for the challenges involved in Rust’s design.

Being compiled is not, itself, an obstacle

In your question, you write

I know I'm not that strong with compiled languages, but shouldn't we be able to do something like so?

but the fact that Rust is compiled has little to do with it: lots of languages are compiled. Haskell, which Rust takes a great deal of inspiration from, is compiled to native code in a very traditional way, and it supports comprehensions with great generality. But Haskell, like most programming languages, is not a systems programming language. After all, systems programming is a fairly niche (albeit very important!) domain, so most languages do not choose to expose all of this complexity to the programmer.

But in Rust, asking such things of the programmer is not just expected, it is necessary for the domains Rust was originally designed to target. Given that, avoiding list comprehensions helps to keep the language more orthogonal:

  • Constructs in the language that allocate memory are kept few and controlled, so lifetimes are well-defined, and the user has explicit control over where the memory goes and how long it lives.

  • Other features of the language, such as for loops, can be used to fill in that memory, which can be done in a much more regular way. This avoids coupling allocation and mutation, which permits greater flexibility.

  • In cases where abstractions that combine these concerns are useful, higher-order functions like map and filter are readily available, and they can be given precisely the types and lifetime signatures they need, on a case-by-case basis.

Features must pay their weight

In spite of all these challenges, Rust includes many language features that deal with the complexity. In addition to the most obvious example of pointers and references themselves and the presence of the borrow checker, other examples include closures (with their complement of Fn, FnMut, and FnOnce traits) and async/await (with its careful desugaring into a state machine).

Indeed, several of the pieces one would need to implement comprehensions already exist. Much as Rust has the IntoIterator trait for polymorphically injecting values into an Iterator, it also has the FromIterator trait from extracting results. Together with the Iterator methods collect and collect_into, it is possible to achieve many of the same things one could achieve with comprehensions using the Iterator API.

Although it would be possible to specify comprehensions in terms of a desugaring to these APIs, baking something into the language comes at a high cost, and it is not done lightly. Using collect and collect_into reuses existing language features to accomplish the same end, and some of those (such as explicit type disambiguation using the “turbofish” ::<> operator) would not come quite so automatically were comprehensions to be made a language feature.

tl;dr: The design of certain things in Rust is not easy because it expects you, the programmer, to make more choices, and making these choices available is the purpose of the Rust programming language.

$\endgroup$
12
  • 1
    $\begingroup$ The Iterator and FromIterator traits already give programmers control over all of these things. I don't see why list comprehensions would need separate APIs. The only substantial difference I can see between a list comprehension and .map(...).collect() is that in .map(...) you're explicitly passing a callback function, whereas in a list comprehension it is less obvious whether the term x + 1 is evaluated in a different stack frame. $\endgroup$
    – kaya3
    Sep 1, 2023 at 12:34
  • 2
    $\begingroup$ This is wrong, at multiple levels. First of all, while Rust does allow manual memory management, the abstractions one typically use automatically handle it, so that few developers worry about when and where memory is allocated/deallocated while writing Rust. A bit more than in Haskell -- due to lifetimes -- but not overly much. Certainly not as much as in C. (cont.) $\endgroup$ Sep 1, 2023 at 13:06
  • 4
    $\begingroup$ (cont.) Secondly, as mentioned by kaya3, the collect() call allows collecting all elements yielded by an iterator into a collection. let new_arr: [_; 5] = my_arr.into_iter().map(|x| x + 1).collect(); is a "list comprehension" in Rust. It performs all allocations necessary... it's just lacking the sugar. $\endgroup$ Sep 1, 2023 at 13:07
  • 1
    $\begingroup$ @JaredSmith Sure, they’re not fundamentally different. But closures are *essential*—they cannot be implemented as a derived concept—so it makes sense to work them out in spite of the challenges. And indeed, closures are very complex in Rust, between Fn, FnOnce, FnMut, and function pointers. Async/await is another feature with quite a lot of complexity that was deemed worth the effort of implementation. But the question asks why Rust chooses not to support list comprehensions, and the choices one would have to make in doing so are a substantial inhibiting factor. $\endgroup$
    – Alexis King
    Sep 1, 2023 at 17:03
  • 3
    $\begingroup$ @JaredSmith Well, I did say they are not fundamentally incompatible. :) But you’re not the only person to have been confused in that way, so perhaps I ought to edit it to be a little bit clearer. $\endgroup$
    – Alexis King
    Sep 1, 2023 at 17:05
15
$\begingroup$

Python List Comprehensions

In Python, there are 3 blessed collections -- list, set, and dict -- for which syntactic sugar built into the language allow creating one, or the other, directly. The List Comprehension syntax available in Python allows creating either:

new_list = [i + 1 for i in old_list]
new_set = {i + 1 for i in old_set}
new_dict = {k:v+1 for (k, v) in old_dict.items()}

This is great when you use them... and less so when you'd like another.

Equality: no second-class citizen in Rust!

Rust is a Systems Programming Language.

In Python, most developers do not know, nor care, exactly how dict works under the hood: what's the memory layout? The type of hash-map used? The hashing algorithm used?

Systems Programmers, however, do care. They may care due to performance reasons, they may care due to the lack of memory allocation in their domain (embedded, kernel). This means that unlike Python developers, they are much more likely to need specialized collections, rather than the run-of-the-mill standard collections.

And therein lies the rub. The thought of all those specialized collections being second-class citizens, and much more awkward to use than 3 blessed standard collections, did not sit well with Rust developers.

Therefore, instead of creating a list-comprehension syntax that would only be usable with 3 standard collections, they instead focused on making the operations you are looking for work with any collection via Rust's powerful trait system:

  1. The Iterator trait allows iterating over anything, and provides many way to filter/transform the elements being yielded.
  2. The FromIterator trait allows any type implementing it to be built from an Iterator.

Therefore, it is perfectly possible to create one collection from another:

let new: Vec<_> = [1, 2, 3, 4, 5].into_iter().map(|i| i + 1).collect();
let new: VecDeque<_> = [1, 2, 3, 4, 5].into_iter().map(|i| i + 1).collect();
let new: LinkedList<_> = [1, 2, 3, 4, 5].into_iter().map(|i| i + 1).collect();

Note: It is not possible to directly create an array due to fallibility concerns, though maybe one should submit a RFC to allow collecting into a Result<[_; _], _> as that would handle the fallibility concerns.

Iterator chains are very powerful: take, skip, filter, filter_map, enumerate, ... so that list comprehensions would be a step down (they can only map and filter) in many cases.

Is it worth having special syntax for the easy case? Well, so far, the answer from Rust developers has been that they don't feel it would be.


On a tangent, the same iterators chains are available for extending an existing collection with more elements:

my_collection.extend([1, 2, 3, 4, 5].into_iter().map(|i| i + 1));

The extend method takes any iterator yielding elements of the same type as the collection.

Macros!

It should also be noted that Rust has macros.

A quick search on crates.io reveals that some Rust users felt the need for quick & easy, and they went ahead and created macros for it, such as the list_comprehension_macro crate (and a bunch of others). From the README:

let even_squares = comp![x.pow(2) for x in arr if x % 2 == 0];
let flatten_matrix = comp![x for row in arr for x in row];
let dict_comp = comp!{x: x.len() for x in arr};

This, in turn, is also a disincentive to include any such functionality in the standard library, as it's easy enough to make your own, or use a 3rd-party library.

Note: a somewhat minimal standard library is also an explicit goal of Rust developers.

$\endgroup$
11
  • 1
    $\begingroup$ So the Macros demonstrate that for/list comprehensions in Rust are possible - in contradiction to the original question and answer langdev.stackexchange.com/a/2851/2163 $\endgroup$ Sep 1, 2023 at 14:03
  • 1
    $\begingroup$ There is also no need for a fallible version of the FromIterator trait, since Result implements FromIterator in a suitable way; see here and this prior feature request. $\endgroup$
    – kaya3
    Sep 1, 2023 at 14:11
  • 4
    $\begingroup$ @ChristianLindig: It's possible indeed, just not built-in. $\endgroup$ Sep 1, 2023 at 15:02
  • 1
    $\begingroup$ I'm not following the argument about blessed types. The generic "comprehension" form in Python is the generator expression that one can throw into arbitrary constructors, e.g. deque(i + 1 for i in old_list). [ ]/{: }/{ } are sugar for list/dict/set constructors by themselves. Semantically, a list/dict/set comprehension is just combining an already blessed constructor with a generator expression. Comprehensions aren't what blesses list/dict/set, nor is blessing list/dict/set needed for comprehensions. $\endgroup$ Sep 2, 2023 at 13:26
  • 1
    $\begingroup$ @ChristianLindig the big difference between a core language feature and a macro in a library is that people expect core language features to work everywhere. A macro in a library (potentially even the standard library, since rust's standard library is now split into multiple sections with different assumptions) has fewer such concerns. $\endgroup$ Sep 6, 2023 at 16:06
6
$\begingroup$

Note: this answer mainly addresses the question of why for loops are not list comprehensions in Rust. The question title was edited later to be about having list comprehensions at all.


Why don't for loops evaluate to arrays?

The feature you're looking for is called list comprehensions. For example, Python has them:

new_arr = [x + 1 for x in my_arr]

Note that even in languages which have this feature, it is not the same as a for loop:

# does nothing
for x in my_arr: x + 1

# syntax error
new_arr = for x in my_arr: x + 1

This makes sense because a for loop is a statement written for its side-effects, whereas a list comprehension is an expression used to generate a list, and the individual terms in the list comprehension are generally recommended to have no side-effects. So these are two different language constructs for two different purposes and there is no need to unify them.

The fact that Rust is an expression-based language (i.e. for loops and blocks are expressions which produce values) doesn't change this; the vast majority of the time you use a for loop, it's not to create a new mapped array, so it would be strange (and possibly wasteful) if this was the behaviour of all for loops.

It would also be problematic for a statically typed language to make every for loop a list comprehension. For example, if the for loop's type was an array or some other sequence, then the two branches of the conditional expression like this one would not share a type:

if xs.is_empty() {
    // type is ()
    println!("No elements");
} else {
    // type is an array of ()
    for x in xs.iter() {
        println!("{x}");
    }
}

So it's better in the much more common case, for the type of for expressions to be (). If you want a mapped array then you can still achieve that by different means.


What about something other than for loops?

So then, why doesn't Rust have a different language construct for list comprehensions? Well, you can already use iterators, .map() and .collect() to build a mapped data structure of any type implementing the FromIterator trait, so the only thing that list comprehensions would add would be syntax sugar for those standard library features.

Such syntax sugar might be valuable, but Rust may choose not to have it because of the language's philosophy that the standard library should not be privileged over third-party libraries any more than necessary; and because syntax sugar can be added by third-party code using macros (e.g. this crate) anyway if some users want it.

$\endgroup$
1
5
$\begingroup$

Maybe because I am not a functional language guy, I don't see at all your proposed syntax as obvious. But if you were to submit an RFC proposal with such thing, the first question that comes to my mind is: what should be the type returned by for?

Neither of the obvious choice seems good:

  1. it can't return an array, because arrays have fixed size, and for iterates over a type implementing the trait Iterator, but a generic Iterator doesn't know its size at compile time. Nor at runtime. Heck, it doesn't even have to terminate, it is perfectly valid for an iterator to be infinite (turning a for into an infinite loop). Thus, array is out of question.
  2. it could return a Vec: if the compiler detects the for return value is being used, it creates a Vec and inserts the output of every iteration. But Vec is a high level library class, it doesn't bode well for a fundamental language construct like for to create and return a Vec. But what is worse is that Vec is not even available for every rust program, because it allocates memory. If you mark your program as #[no_std] (which is used for bare metal programming and other limited resources environments), it won't have a default global allocator, and Vec and anything that allocates memory won't be available.
  3. it could return an inferred FromIterator type, like Iterator::collect() does. Then you could use any container you like, including some that doesn't need to allocate for #[no_std] environments. To avoid allocation, each for iteration would have to be executed on demand from the FromIterator::from_iter() function. I think this completely subverts the distinction between for and an iterator (for has to be effectively turned into an iterator for this to work), and violates the principle of least surprise. We expect a for to be greedy and execute right now, while we expect an iterator to be lazy and only produce the next element when called, if ever. In this case, each for iteration would have to be called from the FromIterator implementation, and executed in between whatever code is in there, including the possibility of side effects, because this can be an user supplied function. It can even avoid running the iterator entirely, and then your for simply does not execute.

Given the return type difficulty, the non-obviousness of the construct (for me at least, but I believe I am representative of the average rust programmer), and most importantly, the readily available alternatives of my_arr.map(...) or even any_iterable.iter().map(...).collect() (which is basically solution 3), I doubt this was ever seriously considered.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .