18
$\begingroup$

As a starting developer, I worked with Pascal and Delphi. Later on, I came in contact with C++ and Visual Basic. All those are native languages, which means that their compiled binaries were executables, dynamic libraries (*.dll or *.so), to be launched on the underlying platform (Windows, UNIX, ...) and the process of compiling was not reversible.

Later on, I heard about Java. A strange language, in the sense that it does not get compiled into binaries which run directly on the platform, but on some "runtime engine", and those could run the same compiled Java sources (so-called bytecode) on whatever platform. Another weird thing is that the compilation process was reversible: Java compiled binaries can be decompiled!

Some years ago, I worked (for a very short and difficult period) for a firm, where programming was done in ABL (or Progress, as they called it). That language had similar properties as Java: working in a runtime environment and being decompilable.

Just out of curiosity, I wonder if both properties are linked, hence these two questions:

  1. Is it possible to have a computer language, being both decompilable and not needing a runtime engine?
  2. How about a computer language, launched using a runtime engine but not being decompilable, would this make sense?

P.S.1 the ideal tag for this question is "language-lawyer", but this tag only exists on StackOverflow. Is this the right place for this question or do I need to migrate it to StackOverflow?
P.S.2 currently I'm working with C#. This compiles to executables or DLLs, a "decompiler" (JetBrains) exists, but the results are very disappointing, hence my first question.

Edit
This is how I understand a compilation being reversible: Compiling source code is something which is done in several steps, some of those are reversible, some are not, and apparently the first steps are reversible. Binary code being decompilable means that only the first (reversible) steps have been executed. This means that decompilable compilers only go up to a certain depth.
Managed code also means that the compilation does not go up to the final depth.
Hence my (apparently wrong) idea:
Would there be a link between "decompilable depth" and "managed depth"?

$\endgroup$
6
  • 3
    $\begingroup$ "apparently the first steps are reversible" Could you give a reference to where you read this? Even what is typically the very first stage of compliation (tokenizing the input) is not reversible as it will often throw away things like insignificant whitespace etc. $\endgroup$ Feb 22 at 16:16
  • 7
    $\begingroup$ This question seems more appropriate for Programming Language Design and Implementation. $\endgroup$
    – Barmar
    Feb 23 at 15:24
  • 3
    $\begingroup$ Not all pascals compiled to machine code. Some of the early pascals for microcomputers (UCSD( compiled to pcode. This was then run by a pcode engine at runtime. Compiling to machine code or byte code is choice that can made by implementers. It is not fixed per language. $\endgroup$
    – Walter
    Feb 23 at 18:12
  • $\begingroup$ @Barmar: thanks a lot for your comment, I didn't even know this website. However I believe I will just leave my question here: in a matter of days, more than 10 upvotes, thousands of views, ..., I have the impression that this community has no problem with this question being here. $\endgroup$
    – Dominique
    Feb 24 at 9:13
  • $\begingroup$ IMHO any binary could be decompiled... the problem is that you will not end up with the same source code. For 2 reasons: 1) In most cases source code <-> binary is a many to many relationship 2) You do lose some information about the source code during normal compilation, you can recoup some of it or use a lot of heuristic to infer some of that but the end result probably won't be similar to a human generated source code... then again maybe in this day and age it's possible to train a huge LLM specifically to convert "naive decompiled binary" to "human C/C++"... $\endgroup$
    – Bakuriu
    Feb 24 at 13:42

8 Answers 8

46
$\begingroup$

The main reason why Java (and in theory C# as well) is more easily decompilable than languages which get compiled to native machine code is that Java and C# both provide a feature which makes this easier for decompilers: Reflection.

Reflection makes it necessary that the types in - let's say Java - binaries provide all the metadata about their fields, method names, method signatures, interfaces, access modifiers and so on. Hence, the type structure cannot easily be "optimized away" at the bytecode level, and a decompiler can utilize this structure.

Of course, though their binaries are not "aggressively optimized", Java and C# programs can be executed quite efficiently. That's because they are usually processed by efficient Just-In-Time compilers - which are part of every runtime. Note Reflection does not require the compiler to keep code comments in the binary, so this part of the source code is usually lost after compiling to byte code and cannot be reconstructed.

So in short, Reflection makes it likely that the compiler will produce binaries which are "not-too-optimized", in such a way that they can be still decompiled, and a runtime with a JIT is then required to make those programs work efficiently. Of course, this design of a language implementation is not mandatory, but seen frequently today.

Would it be possible to have a language "being decompilable" and not needing a runtime engine? Sure, that's simple. As a thought experiment, someone could create a C or C++ compiler which adds enough metadata into an executable binary to make decompilation easy. It could, for example, insert the whole source code into an unused data section.

Indeed, most modern compilers (not just C or C++) can actually add such metadata to their output - when you compile the code for the purpose of source level debugging (inside or alongside to the compiled binary). That is, however, a feature of the compiler/debugging environment, not a property of the language, and this feature is often used in combination with the disabling of certain optimizations which would defeat decompilation.

Now to your second question about programming languages / ecosystems which require a run-time but are not as easily decompilable. That's also not just theory: you may heave heard of tools like Obfuscators for Java and C#. Though not perfect, they can it make pretty hard to decompile byte code into a form a human can efficiently work with. Those tools, however, will often conflict with the use of Reflection - which is IMHO not very astonishing.

There are also older language implementations like UCSD Pascal, where the compiler created byte code (known as P-Code), so a run-time system was definitely necessary. Still, AFAIK UCSD Pascal did not provide Reflection, hence I would not expect a decompiler for that language to work as well as a modern decompiler for Java.

$\endgroup$
8
  • 4
    $\begingroup$ Another factor for Java at least is the debugging information that is typically included in the byte code. By default, line numbers and source file information is included. $\endgroup$
    – JimmyJames
    Feb 22 at 18:10
  • $\begingroup$ @JimmyJames: indeed, the more metadata a binary includes, the easier decompilation gets, regardless of the language. $\endgroup$
    – Doc Brown
    Feb 23 at 8:43
  • 1
    $\begingroup$ It's the runtime reflection that's an issue. Compile time reflection would of course have less of that problem. $\endgroup$ Feb 23 at 9:51
  • 1
    $\begingroup$ @AykhanHagverdili: Indeed. Specifically, while compile-time reflection can be used to generate information used to reflect at run-time -- think serialization -- only the information that is strictly necessary -- only fields, for example, and only for those classes which opted in -- would be encoded, resulting in an overall decrease of metadata. $\endgroup$ Feb 23 at 10:09
  • 1
    $\begingroup$ @AykhanHagverdili: I have no idea why you use the words "issue" and "problem" here. $\endgroup$
    – Doc Brown
    Feb 23 at 12:33
19
$\begingroup$

You can "decompile" any language, but decompilation never gives you the exact same code that was typed. It's not a magic undo button and the results are often not very human readable.

Compiling to a higher level language such as Java bytecode or Common Intermediate Language can preserve more of the original code structure than machine code. So decompiling these can generate more human readable source code.

Whether a particular decompilation "works" is probably just a human judgement on whether a particular person can understand the decompiled code and find what they were looking for.

$\endgroup$
4
  • 6
    $\begingroup$ "You can "decompile" any language" – In fact, "compilation" is the process of translating a program from language A to language B, preserving its semantics, so "decompilation" is just another word for "compilation" where the user has a certain perception of the readability of languages A and B, namely that B is more readable than A. $\endgroup$ Feb 23 at 17:44
  • $\begingroup$ hmmm, unless the language hasn't been compiled yet. $\endgroup$
    – Ewan
    Feb 23 at 18:10
  • 2
    $\begingroup$ There are languages with "perfect" (or at least nearly so) decompilation. The Papyrus scripting language used inside of multiple Bethesda games has a 1:1 compiler to bytecode, and so it can be decompiled into exactly the same source (minus the comments, because I suppose we can't have everything, and some of the syntax might be "regularized" to a limited extent). $\endgroup$
    – Kevin
    Feb 24 at 21:24
  • $\begingroup$ If you assume that your executable was produced by a known compiler X where every detail of the implementation is known, compiling an unknown program Y, then finding Y could be very hard. $\endgroup$
    – gnasher729
    Feb 25 at 22:16
14
$\begingroup$

Your specific questions are not really answerable, because your question contains a number of assumptions that are not true (or at least, not entirely accurate).

the compilation process was reversible: Java compiled binaries can be decompiled!

Yes, Java class files can be decompiled. But the process is not reversible - you are not guaranteed to get back the same Java code as you started with if you go through a compile -> decompile loop.

A computer language, being decompilable and not needing a runtime engine, is this possible?

This depends what you mean by "decompilable". If you mean "definitely get back the original source code", then no - but then that's not possible for Java or to a very real extent any other compiled language either. However, there are plenty of tools which will make a good attempt at decompiling your native code binaries - e.g. Ghidra.

A computer language, being launched using a runtime engine but not being decompilable, would this make sense?

Again depends what you mean by "decompilable". There is theoretically no difference between x64 opcodes and JVM opcodes - they are both just instructions that specify how to execute a program. Sure, you typically are executing x64 opcodes on an actual physical CPU and JVM opcodes in some kind of runtime environment, but there's nothing that stops you having an x64 runtime environment or a physical JVM CPU. If I've got the opcodes, I can attempt to decompile it to human-readable source code. I might not do it very well, but fundamentally the process doesn't change based on the instruction set you give me.

$\endgroup$
12
$\begingroup$

Managed languages, like .NET and Java, use an intermediate code format specifically designed for cross-compilation, since the whole point of the intermediate language is to be cross-compiled to native code for actual execution. This typically includes sufficient metadata so that types, methods, and instructions can easily be located.

Decompilation is a kind of cross-compilation, the target language is just source of machine code, so that is also made easier. But do note that the decompiled code may bear little resemblance to the original source.

Also note that intermediate code do not necessarily need a "runtime". It is often possible to cross compile it to native code as a separate step, usually called "Ahead of Time" compilation, to contrast with the more common "Just in Time" compilation.

Native instruction sets often contain features like indirect jumps. And this creates a major problem for decompilers, cross-compilers, etc. This is where the address of the next instruction depend on the value of a variable, and there is no way to know the value of a variable without actually running the program. The result of this is that you cannot know what bytes represent instructions, and what is data, at least not without some kind of extra metadata. This is why you usually need some level of emulation to run code for another processor architecture.

You could produce an native instruction set designed to be easy to cross-compile/decompile. But that might or might not make sense from a business perspective. You could also design an intermediate language that could not be cross-compiled, but that seem to go against the purpose of an intermediate language.

$\endgroup$
2
  • $\begingroup$ Also metadata is needed to enable garbage collectors to understand the layout of memory, a decompiler makes use of this metadata. $\endgroup$ Feb 23 at 12:25
  • $\begingroup$ Many compiler optimizations that have side effect of making decompiler hard are not done until the code generation passes of a compiler. By definition full code generation is not done when outoutting intermediate code format. $\endgroup$ Feb 23 at 12:30
5
$\begingroup$

Your labeling of "native languages" and "decompilable" languages is largely artificial and not really correct.

This is similar to the false distinction between "compiled" languages and "interpreted" languages.

Any language can be compiled to native machine code. Any language can be interpreted. Which one you have is really an implementation detail.

For instance, C is generally considered to be a compiled language, but there are C interpreters that can directly run C code (slowly) without compiling it. Most BASIC languages were designed to be purely interpreted, but there were compilers for basic that would generate native machine code to make it faster (and difficult to decompile).

Java is generally both compiled and interpreted. You compile source code to a "byte code" that is really machine code for a processor that is then emulated by the JVM. But real Java CPUs exist that can directly run this machine code! And modern JVMs actually convert the bytecode to native machine code at runtime (just-in-time compiling) to get extra speed. As pointed out by others, Pascal was generally considered a compiled language, but many Pascal implementations compiled to a portable bytecode that was machine code for an emulated CPU, so that porting Pascal was just a matter of porting the CPU emulator. But again, this "emulated" CPU was actually built in physical hardware at some point.

And then there are languages like Julia, which is both compiled and interpreted. (Julia is not unique in this.) Basically, what this means is that while most aspects of the compiler output is code that can be run directly on the native CPU, new code can be generated at runtime and then run. To make this work, compiler itself must be part of the runtime. With this model, even languages designed to be purely interpreted can still be compiled!

Similarly, your distinction of languages that can be decompiled is artificial. Any language can be decompiled, but the result is likely not identical to the original source and may not be human-readable without a lot of work to add things like variable names. The only languages that can truly be reversed are tokenized (like some early BASIC languages) rather than compiled, or include enough information (for introspection or debugging) to reverse the compilation process or are not optimized at all. If there is any optimization, there is not a 1:1 correspondence between the source and the compiler output, and likely things like internal variable names are completely lost as they are assigned to CPU registers that might be reused for multiple variables in different sections of code. Generally, if the compiler emits native machine code on a CPU with limited registers, this will always be the case even without an optimizer.

$\endgroup$
2
$\begingroup$

Any compiler converts some kind of code into other kinds of code. Like C to machine code or Rust to C. You typically move closer to what a machine can actually execute.

Java has two compilation steps. One from humanly readable source into a high level byte code, and one from that byte code into actual machine code (the latter happens on runtime so is rarely visible). It was decided back then to put the efforts on the second step which is now paying off. Even though Java was mocked back then for being slow, Suns focus on correctness and robustness is now a major reason that Java is a frequent choice of what runs on the servers even though .NET Core is catching up. As Java is not slow anymore, Oracle is adding functionality that allows you to execute the humanly readable source code directly at runtime!

If you as a Java programmer do not want your code to be reverse engineerable, you can choose other forms. There are obfuscators like proguard which changes things to make it hard, and compilers like Excelsior JET and GraalVM that can produce the final machine code ahead of time. Typically in my opinion there is rarely a need, as if you are in a situation where this matters there are usually already contracts in place that cover the way you can use the code.

$\endgroup$
2
$\begingroup$

First you need a good understanding of the difference between compiled and interpreted. This is not a matter of language, but of distribution and process. If what is distributed and processed (executed) is what the developer wrote, then it’s interpreted. If what is distributed has been transformed (ignoring stripping of white space and comments and possibly obscuration), it’s compiled.

With interpreted distribution, there is nothing to decompile, what is available is the original.

With compilation, the transformation is almost certainly going to be lossy, ie you can’t go back and forth between the two without loss because that’s not a design goal and adding the ability to do so would generally be a waste — you are looking for the new format, not to allow someone else to get the source you already have.

That said, when the destination is an intermediate format, one which is itself expected to be compiled/interpreted , it tends to keep a similar structure and the same function names. Global optimization and inline methods are generally not done. All of which means that the compiled code is still somewhat similar to the original code.

Despite this similar structure and at least some names being retained, it’s still intended to be a one way process, and is going to be lossy, just not as lossy as would be the case if it was being compiled to a machine code.

One other point, sometimes compilers generate debugging information that allows associating the compiled code back to the original source code.

Anyway, to sum it up, languages that compile to intermediate formats (managed languages) can generally be decompiled back to something closer to the original more easily than those that aren’t, but this isn’t a hard and fast rule and depends upon the purpose the intermediate language than the fact that it is an intermediate language.

$\endgroup$
1
$\begingroup$

A computer language, being decompilable and not needing a runtime engine, is this possible?

Yes. In the stage where the compiler creates the assembly (or later machine code), it could easily inject special bytes which are skipped during execution, but encode the original source material in a format that makes reconstructing the source code possible. As a crass thought experiment, it could simply append the original source code as-is within the executable in some kind of data structure that is not actually executed.

This would of course take a lot of effort (not necessarily during runtime, but to come up with a sensible, efficient encoding scheme).

How to inject arbitrary bytes into an executable is a minor implementation detail, but a trivial way would be to just put them in there and skip them through a simple (relative) JMP instruction, depending on what the target architecture offers.

A computer language, being launched using a runtime engine but not being decompilable, would this make sense?

Yes, for sure. It would not only make sense in whether it would be technically possible, but also for example to make it harder to decompile (e.g. obfuscation etc.). In this case you would do the opposite of what I described above, you would skip injecting some kind of meta information into the bytecode. Then decompiling said bytecode would be as difficult or easy as decompiling "actual" machine code, and would require deep knowledge about how the compiler works - even if is goes heavy on the optimization.

N.B. there is not really that much difference between a native compiler targeting a CPU directly, and a bytecode compiler. The bytecode is simply the "machine code" of the runtime interpreter. Yes, the details are of course wildly different (since the runtime and the compiler are probably designed and developed in tandem, and can be targeted as a much more high-level architecture than a CPU, many things can be designed in that would be tough for a physical CPU architecture - for example a class system, the bytecode stored in a tree instead of sequential OP codes etc.), but there is no fundamental difference regarding decompilation.

$\endgroup$
2
  • 1
    $\begingroup$ The "special bytes" approach doesn't even need a complicated data structure. Common executable file formats tend to ignore any junk data past the end of their structures, so the source code can simply be appended to the target file. Even better, if you put the entire source code in a ZIP file first, extracting the source code becomes very easy, because the ZIP format works "back to front" and such is extractable even with unreadable junk (aka the actual executable) prepended to it! I learned this works from LÖVE, where games were generally distributed in "fused" form. $\endgroup$
    – Jasmijn
    Feb 28 at 20:36
  • $\begingroup$ Very correct, @Jasmijn, thanks for the thoughts! $\endgroup$
    – AnoE
    Feb 29 at 8:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .