Yes, a very loose summary of a system I worked with (and implementing) pre-2007:
The main part was an assembly language designed with a view to implementing Java. Or maybe refined for Java, I don't remember: that part happened before I joined the company. Think of an assembly language with a lot of macro facilities built in, including macros that make virtual function calls and some other relatively high-level concepts. All this targeted a virtual instruction set that would later be translated to actual machine code (not a million miles from the idea of LLVM instructions, but invented before LLVM itself existed).
So, all essential JVM opcodes were available as assembler macros or function calls. At the same time, by which I mean, from the same source code written in the same language inside the same function as these Java operations, the programmer also had direct access to the kernel calls of its non-protected mode OS. And also to a stdlib style set of functions. The stdlib was mostly used by the system's C compiler when it generated the backend code, but there's nothing to stop anyone calling them who is writing in assembly.
If you stored a pointer to a Java object into a marked region of memory (the stack or a Java object, basically) then it would be GCed normally (normally for Java, I mean: mark sweep). You could also allocate unmanaged memory just as easily with
free themselves, or with similar-style kernel functions. I'm pretty sure there was also some facility to help with reference counting, although I don't remember exactly how that worked. Everything you can do in Java translated reasonably obviously to assembly code, it just took longer to write.
So, when writing code you genuinely made a choice for each piece of memory whether to allocate it as a Java object (in which case it would be GCed provided you didn't mess up and fail to store the pointer anywhere), from the kernel (in which case you were responsible for freeing it, via the assisted refcounting if you wanted it), or from libc (in which case you were broadly responsible for freeing it, but as a concession and to support easier porting of existing C programs, there was a concept of a process, and processes did keep records of all their
malloc allocations and other C-style resources, to clean up on process exit).
It therefore had more mixed modes of resource management than any one piece of code really wanted to use: you'd mix at most two of the three in practice.
This system was never publicly available as a programming environment, although as a Java implementation it did appear in some embedded devices, including early HTC Java phones. It also made it onto some other devices without the Java parts: it was quite modular in that sense and you could build a stripped-down version of the OS that just didn't include anything you didn't call.
And fundamentally, it did what you ask for but probably not what you want. It didn't have the "ease of use" of Java, because you literally had to think, "OK, is my next virtual call made through an interface or through a class", because invoke_virtual and invoke_interface were different macros. It had a language restriction that each instruction had at most one side-effect (counting "call out" as one side-effect): you could write
a + b + c if those were integers, but you literally couldn't write
a() + b() + c() if those were function pointers. You couldn't even write
total += a(). You had to write the equivalent of
result = a(); total += result. You couldn't write
a + b + c if those were strings, either: it would be probably 4 lines of code to create a StringBuilder from
a, then append
b, then append
c, then create a string from the result. When you did that, all that memory would be GCed. But the assembler absolutely did not type check your code: but you didn't ask for type checking, you asked for mixed memory management ;-)
One sense it probably didn't do what you're asking: you could mix the modes, but not for the same types. If you want a GCed array of integers, fine, use a Java array (and of course you could get the base address to do direct memory access to it as a buffer). But if you want to manually manage a Java object, you can't: there was no legitimate way to call "free" on a java.lang.String on grounds you happen to know you're the last remaining reference to it. If you want to do that you need to allocate a buffer for a nul-terminated byte array from the kernel, instead, and then it wouldn't have all the convenience methods of String. Hello,
So there's a reason it was never intended as the new general-purpose programming language for the world. But for relatively high-end (for the time) embedded systems programming, and for accelerating Java code before Sun ever released a JIT for mobile devices: it wasn't bad. We beat Sun on pretty much all the benchmarks at the time. You could take Java code, figure out which bits "really needed" GC and which could be manually managed, and rewrite the Java code to run faster and occupy less RAM. We literally did that for some critical parts of the Java standard libraries, and left other parts as the off-the-shelf Java implementations.