Your question implicitly covers three different comparisons:
- Different execution models used in implementations of a language. For instance, the same Python code can be executed using either CPython or PyPy.
- Differences within an implementation between different settings. For instance, systems often combine an interpreter and a JIT compiler, allowing comparisons "with and without JIT".
Rather than thinking of different "implementations of JIT compilation", it is perhaps more helpful to think of JIT compilation as a category of techniques. Different languages, and different implementations, will apply those techniques differently, so it's hard to put a single value on "the benefits of JIT".
Compilation vs Interpretation
Discussions of execution models generally begin with the traditional distinction between a compiler which takes a unit of source code, produces equivalent processor instructions, and discards the source code; contrasted with an interpreter which reads directly from the source code, translating and immediately executing one instruction at a time. In practice, the picture is considerably more complex than that.
Firstly, there is a spectrum of how much is compiled. Modern compilers proceed through multiple stages of intermediate representation - for instance, a stream of tokens, an abstract syntax tree, a target-independent bytecode, and then the target instruction set. An interpreter can take over at any stage of this process - some BASIC implementations replace keywords with binary tokens, then interpret one instruction at a time. One of the most common current models is to compile source code to some level of bytecode, then interpret that.
Secondly, there is variation in when compilation happens. Compilation may be a separate step explicitly invoked by the user, or it may happen implicitly when a particular module of source code is loaded. The result may be stored on disk, kept in the running process's memory, or cached in shared memory owned by the run-time.
- Java source code is compiled explicitly by the user to JVM bytecode, and saved to disk. The bytecode is a low-level instruction set for a stack-based virtual machine, and can be produced for languages other than Java.
- The PHP reference implementation is the Zend Engine, which compiles PHP source code one file at a time into "op codes"; the compiler is generally invoked the first time a file is included and the result cached in shared memory by a module called OpCache. The op codes are high-level instructions closely tied to the language, which are then executed in an interpreter as the program runs.
This leads us to "dynamic compilation" and "just-in-time compilation". There's a fascinating look into the history of the concepts here: John Aycock. 2003. A brief history of just-in-time. ACM Comput. Surv. 35, 2 (June 2003), 97–113. The term "JIT" was popularised by Sun's "HotSpot" implementation of Java, and is generally used to refer to systems where:
- Machine code is generated while the program is running
- The compiler optimises the result with information obtained from the running program
(Note that while the PHP Zend Engine discussed above could be described as "compiling just in time", it is not normally referred to as a "JIT compiler"; we could instead label it a compile and go system.)
The fact that the program is already executing rather than about to execute makes additional information available to the optimiser, such as:
- Which path through conditional code is most often taken
- Which types are most often passed to a function which takes a dynamic type as input
- Which polymorphic methods / dynamically dispatched functions are frequently resolved in a particular context
Theoretically, producing optimised machine code in advance can always give better run-time performance than producing that same machine code while the program is running, because the optimisation and compilation itself take time. This can include feedback from a previous execution of the program, as in profile-guided optimisation. JIT compilation is mostly useful when it is impossible, or very difficult, to produce that "perfect" code ahead of time; for instance:
- The implementation stores an intermediate representation rather than a native executable by design; e.g. Java's JVM (which provides portability) or .Net's CLR (which provides secure interoperation between source languages)
In such implementations, you can see the impact the JIT is having on a particular scenario by disabling it and forcing everything through the interpreter; but that doesn't mean a different interpreter (e.g. with a different form of bytecode) couldn't be just as fast. In the case of PHP, the improvements to the Zend Engine in PHP 7.0 had a bigger impact on many workloads than adding a JIT on top in PHP 8.0.
The reference implementation of Julia uses a slightly different strategy, sometimes referred to as a "just ahead of time compiler". Rather than treating the interpreter as the default, and optimising certain paths during run-time, it focusses primarily on compiling individual functions specialised to particular types. It is just one example of how varied strategies can be, and how hard it is to compare unless there are multiple mature implementations for the same language.