35
$\begingroup$

Chapter 1 of "The Rust Programming Language" (Klabnik and Nichols) says:

[S]ome common Rust packages depend on C code and will need a C compiler.

Why do Rust packages have any dependency on C code? That is, why does Rust code ever need to use something that isn't also written in Rust?

$\endgroup$
11
  • 22
    $\begingroup$ Is your expectation that every layer of the stack should be rewritten for every language (i.e. single-language machines), or are you thinking about the implementation of those languages themselves (in another language versus self-hosted), something specific about foreign-function interfaces & dynamic loading in Rust, properties of C, or something else? The question is fairly broad now and I can imagine e.g. the situation for Rust and Python is different, but there are a lot of question marks going in different directions too — which are the question to be answered and which are giving context? $\endgroup$
    – Michael Homer
    Nov 25, 2023 at 4:50
  • 1
    $\begingroup$ To the extent that the question is that first point and your comment on the answer, I don’t think this is a PL question after all, but something about the history of Unix or computing. Single-language machines have existed in the past but are not common general-purpose systems today, and while e.g. reimplementing GTK in every language seems obviously undesirable it’s also not what you’re asking about. If you can edit it down to a single on-topic question I think there is something answerable in here, and maybe other separate questions too, but I’m not sure they’re what you’re looking for. $\endgroup$
    – Michael Homer
    Nov 25, 2023 at 17:42
  • 3
    $\begingroup$ I think the close reason is wrong, but closing is appropriate: there are just too many different questions here. The answer to "why does Python rely on C?" is trivial: Python isn't compiled directly to machine code, and sometimes you need the performance of something that is. The answer to "why does Rust rely on C?" is probably more about pragmatism of what already exists. The answers to all of your supplemental "is it...?" questions are "yes" for some languages and "no" for others, with a side helping of "because languages are designed by people who have opinions". $\endgroup$
    – IMSoP
    Nov 26, 2023 at 0:10
  • 4
    $\begingroup$ I still think this question should have remained closed, but I have edited it down to one single question that I think was indicated as being the main question. It still doesn't really seem to be about programming language design or implementation so much as overall system architecture and pragmatics, though. $\endgroup$
    – Michael Homer
    Nov 26, 2023 at 1:56
  • 2
    $\begingroup$ You mean phrasing like "depend on libraries written in C"? Probably that's exactly what the authors meant by their phrasing, and didn't consider that it might not be obvious to all readers. $\endgroup$ Nov 28, 2023 at 5:04

2 Answers 2

94
$\begingroup$

Why do Rust packages have any dependency on C code?

Taken literally, this question is not really on-topic: it’s a question about some decisions some people made when they wrote some libraries, not a question about the Rust programming language. By and large, their motivations likely have more to do with the practicalities of software engineering than anything fundamental about programming languages. Ultimately, it’s just plain convenient to reuse C code because there’s a lot of it, it’s particularly easy to call into, C toolchains are ubiquitous, and whatever runtime support it needs is almost certainly already loaded into your process.

Still, if we follow that reasoning just a little bit further, there are definitely a few programming language questions hidden in there. For example, why is C so exceptionally easy to call from other languages? Why does Rust have built-in support for C calling conventions and struct layout but not other languages? This is by no means specific to Rust: in most languages, “foreign function interface” might as well be a synonym for “a way to call into C”, and it is not at all uncommon for C toolchains to show up in unexpected places.

Why is that?

C ABIs

The phrase “the C ABI” is something of a misnomer, as C does not specify an ABI, nor is it really defined at the level that would allow specifying one. Rather, “the C ABI” refers to a general collection of things defined for every combination of architecture and operating system. For example:

  • The calling convention specifies a standard set of rules for how arguments are passed to and returned from functions, the layout of the stack, and what registers are preserved across the call.

  • Datatype size and alignment rules specify the necessary size and alignment of primitive types like int, long, and float.

  • Structure type layout rules specify how composite structures are laid out in memory and how they are passed to and returned from functions.

  • Executable formats define how executables and shared libraries should be laid out so that the operating system knows how to load them into memory, set up their virtual address space, and dynamically link them.

It is easy to quibble about what is and is not part of “the C ABI” for a given platform. Are executable formats really a part of the C ABI, or are they something else? What about thread local variables? Signal handling? Debugging symbols and unwinding information? I’m not going to try to pin down some precise list because there simply isn’t one. What matters is two things:

  1. Where precisely the C ABI and C standard library end and your operating system begins is not at all clear.

  2. It is extraordinarily important that the C ABI for a given platform does not ever change.

These two things are very closely related.

C is your operating system’s interface

As apropos’ answer points out, there is a real sense in which C isn’t (just) a programming language. A staggering amount of essential operating system functionality is not just most convenient to access via C libraries, it’s the only supported way of accessing it. Sure, if you really want, you can find the conventions for making a syscall “directly”, but the idea that syscalls are the interface between userland and the operating system is rather idealistic. For one, syscalls are the interface between userland and the kernel, and the kernel is really an exceedingly small part of even a fairly minimal modern operating system. For another, they are poorly documented and require reverse-engineering on proprietary operating systems, and worse, syscalls are not even stable on Windows.

So it is really rather black and white: the intended way to call into your operating system is to call a C function. In fact, let me make this even more explicit: it is not just the intended way for a human to write a computer program that calls into your operating system, it is the intended mechanism by which every programming language in existence calls into your operating system. On Linux, one can perhaps quibble with this (though I still think it would be quite ill-advised), but on macOS and Windows, you really cannot, so there’s just no way around it—your programming language needs some way to (eventually) make C calls.

That said, if your programming language is interpreted, you basically don’t have to worry about this problem. Whatever language your interpreter is written in already knows how to make the necessary calls, so you can just defer to that. In fact, even a compiled language that uses its own binary object format and does its own linking and loading and has its own runtime can largely get away with not knowing how to generate C calls because the program can always yield to the runtime’s scheduler, which is written in a language that knows how to perform the call into C.

This means that, counterintuitively, being a systems language without a runtime (like Rust) means you actually have to care more about the details of the C ABI on each and every platform because you have to be able to generate code that makes those calls directly. You have to speak the platform’s calling convention, you have to do all your stack layout in a way that’s compatible with the C stack, you need mechanisms for generating code that uses the correct structure layout and alignment rules, and you need to figure out what to do when a function written in your language returns to a function written in C, something that is not altogether obvious when your language provides functionality like exceptions that is not supported by the C ABI.

C ABIs are difficult to separate from the C language

After everything I said above, you hopefully understand why just about every programming language under the sun needs some way to speak the platform’s C ABI, but you might wonder whether calling it “the C ABI” is very useful. You might think it would be clearer if we called it “the platform ABI,” and in fact that is often done! This might lead you to believe that we could dispense with all the programming language parts of C—like type declarations and header files and .o objects—and just focus on the binary interface bits, which really don’t need to be considered through the lens of C at all.

Sadly, if you actually try to implement a compiler, you will rapidly discover that this ABI is painfully difficult to disentangle from the programming language C. In order to make a C call, you don’t just need to know what registers to pass arguments in and what size an int is, you need to know things like struct layout and what it means to spill a struct onto the stack. You also need some way to take the interface specifications provided by operating systems and turn them into a format your compiler can work with, and wouldn’t you know it, those interface specifications are provided in the form of… C header files.

No, really: go down this rabbit hole and it will start to drive you mad. Pick just about any syscall that involves a struct type and try to figure out how to automatically determine how to generate the right code to use it. Consider the case of stat. The man page says it’s in sys/stat.h, but how do we even find where that header is on some arbitrary system? The highest-voted answer to a Stack Overflow question about how to do this on Linux is to call gcc. I assure you it does not get any easier to avoid giving in and calling a C compiler from here.

The reality is that C header files are currently used as the lowest-level interface description language you can get your hands on, and this is a somewhat frustrating choice seeing as it is virtually impossible to parse them without writing a C compiler. But let’s suppose you did that—you deduced all the platform-specific rules for layout and calling conventions, you reimplemented the C preprocessor, and you wrote a fully-featured C parser and symbol resolver that allows you to parse any standard C header file—you will rapidly run into another problem.

The header files on your system are not written in standard C.

C ABIs are difficult to separate from C toolchains

This part can come as a bit of a shock to people, especially since there’s some idea that the header files found on your computer are compiler-agnostic. In fact, if you go digging through your system’s header files, you’ll find an awful lot of #ifdef __GNU_C__ and #ifdef __clang_major__ littered about. In practice, your header files are very much vendor-specific, they just explicitly handle a small list of known vendors.

At this point it may become clear how hopeless this all is. The deeper you look, the fuzzier the line between your operating system and the C compiler becomes. In a way, the entire C toolchain is a part of the operating system, the wildly complex decoder ring you are forced to use if you want to be able to conform to the eye-wateringly elaborate machinations required to call into your operating system. Yes, I realize this might sound hysterical, but consider just how deeply the C toolchain is interwoven with languages that, frankly, would rather have as little to do with it as possible:

There will always be people who quixotically struggle to avoid depending on the C toolchain at all costs, but doing so is essentially hopeless. Even if one were to implement everything required to consume system header files without MSVC, GCC, or Clang, support for those headers is such a moving target that it would likely create a headache-inducing maintenance burden. What’s more, C’s exceptional ubiquity only entrenches it further: when software written in different languages needs to coexist in the same process, the mediator is the collection of languages, tools, runtime infrastructure, and operating system support that we now know under the moniker “C”. Why not something else? Because everything already supports C. So everything has to support C.

To quote the current Twitter bio of JeanHeyd Meneide, current Project Editor for the C standardization committee:

The C Standard Cannot Be Replaced And Will Never Be Destroyed.

Quite.

$\endgroup$
19
  • 4
    $\begingroup$ Funny you mention macOS because that was the huge exception. Classic macOS was not C-based, but Pascal. The Carbon API made it more C-friendly, but most functions utilized Pascal calling, as it was needed for Macintosh Toolbox, and Pascal strings were everywhere. Carbon was finally removed in macOS 10.15 (2020). $\endgroup$
    – user71659
    Nov 26, 2023 at 23:16
  • 8
    $\begingroup$ Old Windows was also more Pascal based than C based -- this shows up in calling conventions with cdecl vs stdcall. C mostly came into the picture with the dominance of UNIX. $\endgroup$
    – Chris Dodd
    Nov 27, 2023 at 0:29
  • 3
    $\begingroup$ To add to the list of OSes: OpenBSD is considering restricting syscalls to libc (its own, obviously) for security reasons. The security aspect, as I understand it, being that it's easier to craft a syscall than it is to craft a call to a function from an ASLR loaded libc, and therefore mandating going through libc will make exploits more difficult to perform. $\endgroup$ Nov 27, 2023 at 8:35
  • 2
    $\begingroup$ On Windows, Microsoft now maintains portable API descriptions (winmd) for Win32, COM and WinRT. The official Rust bindings for Windows are generated from these. $\endgroup$ Nov 28, 2023 at 0:33
  • 3
    $\begingroup$ Re: "C ABI" even though ISO C doesn't define an ABI: The way to read that term is that it's an ABI that supports C, making compatible C implementations possible. A C ABI defines all the functionality needed by C, but usually not much more. e.g. it defines how to call functions with a fixed signature or C-style variadic, and to return one value, possibly an aggregate. And sizes / alignments for the C primitive types. You can have multiple ABIs on a platform, like AArch64 64-bit vs. ILP32. Or some embedded platforms like ARM Linux have had a few different ABIs, not binary compatible. $\endgroup$ Nov 28, 2023 at 5:18
20
$\begingroup$

(note: this answer is addressing a quite different question from the original revision: why one language would have a dependency on another language?)

Is there is something fundamental about C that other languages like Rust simply cannot do?

Yes. C is the language of the Linux kernel. libc is a fundamental library for interacting with the kernel, and reimplementing it in a non-C language would be a bit of a fool's errand (you'd still be interfacing with C regardless). Of course, you don't have to use libc to use Rust for ex. embedded targets, but if you want your code to compile and run and do useful things on Linux, you're going to have to link against it. And to link against it (or anything on Linux), you'll need a C compiler, and have to talk with the C ABI (see Alexis King's answer for much more in depth about this).

Or is it that existing C code is so well written that there's no point reinventing those wheels in newer languages?

Sometimes. curl is an excellent and battle-tested library despite being written in C. I've seen a number of languages - off the top of my head, some libraries in Nim - prefer to link to curl rather than implement the layers of the networking stack themselves. Rust libraries may do the same, though I do not know for sure.

Does it have anything to do with modern programming languages abstracting away certain primitive programming concepts like pointers/memory, thereby leaving some things "impossible" in the newer language?

No. There are certainly things that are impossible to do in some programming languages: an obvious example is nontermination in languages that are total, like Agda (excluding bugs). If you have certain restrictions on how you want to accomplish a task - for example, if you want to return a reference in a language that disallows it like Lobster, or if you want to have a binary under X bytes in a language known for bigger binaries like Rust, then that certainly can be "impossible", but quite a lot of programming languages that abstract over raw concepts like pointers and memory have some sort of escape hatch to allow you to do what you could do in an unsafe language.

I've heard, anecdotally, that some Python packages rely on C code.

A lot of Python packages rely on C code for performance: for example, numpy. This is because Python is Really Goddamn Slow and less so due to limitations of Python.

So perhaps my question is a more generic "Why does language X have any dependency on C code? Why isn't it language X 'all the way down'?"

There are likely more answers to that that I'm unaware of, but you laid out the questions for the answers I know of nicely in your own post.


(I welcome improvements to this answer. I'm not so confident about my knowledge of libc, in particular.)

$\endgroup$
16
  • 3
    $\begingroup$ @apropos You don’t need a C compiler to link against libc, though, so the first paragraph doesn’t actually give an answer. That’s especially the case for Rust which can innately produce C-compatible exports and imports itself. $\endgroup$
    – Michael Homer
    Nov 25, 2023 at 17:24
  • 7
    $\begingroup$ @StoneThrow POSIX defines all system interactions in terms of the C interface and does not define any other kernel interaction, so libc-kernel is one thing as far as userland is concerned. Things like open, which fundamentally interact with the system, need to be accessed that way (or by platform- and even version-specific non-standard interfaces, like numbered syscalls; Go does try to do that). The need to move in lockstep means there are very few even replacement libcs for any platform. $\endgroup$
    – Michael Homer
    Nov 25, 2023 at 17:30
  • 1
    $\begingroup$ "C is special by virtue of being the language of the Linux kernel"; but what does Linux have to do with this programming language's packages? This is an argument the answer could make, but it doesn't, and it doesn't obviously connect to this need for a C compiler - it needs to explain why these are things that the Rust user needs to care about. $\endgroup$
    – Michael Homer
    Nov 26, 2023 at 2:13
  • 1
    $\begingroup$ You can revert it, though the question certainly needed more focus - the mere fact that you're answering five disconnected questions illustrates that! If it can be edited into a focused question that preserves more of what you want, great. The author indicated that they were aiming at why "every layer isn't rewritten for every language", but also said the question was specific to Rust; I'm not sure how the most general version of that couldn't be unanswerably broad. This is one risk of speeding to answer a question you see is being closed, unfortunately... It can be edited further though. $\endgroup$
    – Michael Homer
    Nov 26, 2023 at 2:34
  • 1
    $\begingroup$ @apropos - your "you don't have to use libc to use Rust" link seems to indicate that Rust's need for a C compiler is merely to invoke the linker (presumably ld) -- did I understand that correctly? If so, why use a C compiler to invoke ld? Why not just invoke ld directly? $\endgroup$
    – StoneThrow
    Nov 26, 2023 at 18:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .