Executable files with a bytecode compiler/interpreter

Question

Let's say I have a bytecode compiler. Even though it doesn't generate native machine code, is there a simple way for it to produce native executable files that I can directly run?

One method I've heard of is essentially recompiling the interpreter with the bytecode somehow attached. I think I like this idea, but it would be ideal if the executable file could be used by the interpreter as well as a standalone program. That is, the output of the compiler may be executed or loaded as a module by the interpreter when it executes an import.

I confess I don't fully understand modern executable formats, but I have a basic knowledge of ELF. Something I've thought of is using the “interpreter” field to run a program thst loads the bytecode interpreter and feed it the ELF's text. Is this viable? Is it a bad idea? I'm suggesting ELF because it's a feasible format for compiled modules; GNU GUILE uses it.

Answer 1

It depends on what you mean by "native executable".

If you mean "runs natively on the hardware", then no: if you don't have machine code, then you don't have machine code, and a runtime is required.

If you mean "can be launched from the operating system in the same way that a machine-code-based executable is", then yes - perhaps even trivially. But at this point there isn't a clear distinction between "running an 'executable' file that requires a specific runtime" and "using the runtime to open the file". Which is to say, in Windows I could trivially associate .py or .pyc files with a Python interpreter, type their name as a "command" in CMD or double-click an icon in Explorer, and have the code execute - but few people would call this "making the file executable" even though the user experience may be identical to running a .exe.

Linux, on the other hand, doesn't have the same concept of file association (although a desktop environment might build a reasonable facsimile on top). Instead, you set an executable permissions bit on the file, and then arrange the file format such that it has a signature that is recognized by binfmt_misc.

If you mean "is stored in a container format that the OS commonly uses for machine-code-based executables" - e.g., as ELF on Linux or PE on Windows - yes, it's certainly possible. The .NET platform works this way: code in .NET languages such as C# compiles to bytecode that may end up stored in a file in the PE format - to my understanding, it's stored as "data", and the "code" is a hard-coded bootstrap which makes a system call to locate the runtime and feed the bytecode to it. A similar trick should be equally possible with ELF. (Of course, as you've noted, you could also just put the entire interpreter there, and not worry about needing to find it on the system. Many "Python standalone app builder" utilities work this way.)

On Linux, that approach has the advantage that the system is already configured to recognize the ELF format and "execute" it by loading the machine code.

In any event, if you want such an "executable" to also be usable by the interpreter, you just need to arrange for the interpreter to recognize the "executable format". The trivial example of this is supporting # as a comment character, so that valid source code can have a shebang line; then the interpreter sees ordinary code, while Linux sees an "executable text file" that is "executed" by parsing the shebang to find the interpreter and passing the script to it. A more complex example is Python bytecode: binfmt_misc may be configured to recognize .pyc files according to the header bytes that specify the bytecode version (mapping to a corresponding interpreter).

Related reading: File extensions and association with programs in linux on unix.SE

Answer 2

Here is one way to do it, which is used by the LÖVE game engine:

The bytecode gets stored in a .zip file which gets concatenated to the interpreter.

When the interpreter starts up, it tries to open itself as a .zip file. If it succeeds, it uses that as the program to be executed, otherwise it uses the regular means of finding the program (usually using command line arguments).

This works because native executables generally have their metadata in the front, and will ignore appended "garbage" data, and .zip files have their metadata at the end of the file, and will ignore prepended "garbage" data.

You don't need to wrap it in a .zip if you design the bytecode format itself in a similar way.

For example something like this:

bytecode is <opcodes><strings><debug><metadata>

where

opcodes is X bytes of byte code instructions
strings is Y bytes of strings and other data used by opcodes
debug is Z bytes of debugging information
metadata is <X><Y><Z><magic number> (with each of those fields a fixed number of bytes)

Answer 3

As long as you have a bytecode interpreter implemented, all you need to do is generate a program that's the equivalent of

#include "MyBytecodeInterpreter.h"

uint8_t BYTECODE[] = {
   // insert bytecode here
}

int main()
{
    return InterpretBytecode(BYTECODE);
}

You could even write an actual C program and compile it, if you don't know how to generate executable files directly. Then just link it (statically or dynamically) to your bytecode interpreter, and you've got an instant “compiled” program.

To ensure that the generated executable can be used both as a standalone program and as a module that the interpreter can import, you can export the BYTECODE symbol from the compiled executable. Then modify your interpreter to detect if its input is an executable file (e.g., look for the magic number \x7FELF on Linux or MZ on Windows), and if it is, load the bytecode from the symbol's address.

Sure, it may not run as fast as a “real” compiled program, but it will technically satisfy your requirement of having “native executable files that I can directly run”.

Answer 4

It is certainly possible to write a compiler which accepts bytecode as its source and outputs native code. That's what Java's JIT compilers do, after all.

Writing one that produced standalone executables would be a nuisance (among other things, there's the question of whether to start up Java to run library and utility jarfiles or compiling those too, either inline or linked), but it's really no harder than writing a back-end for other intermediate languages. Though you might actually want a front end which converts it to a more standard IL so you could use existing optimizing back-ends.

If it doesn't already exist (I would expect it does) and you want to attempt it, Go for it. If you're talking about Java bytecodes and are writing your compiler in Java, the BCEL package makes manipulating bytecodes fairly straightforward; that's what we used when implementing compilation to bytecodes in Apache Xalan.

Be aware that, depending on the code involved, this may or may not be able to run faster than what the JIT compiler is already doing for you. That was one of the arguments against offering pre-compilation as a feature of the JDK/JRE.