What I'm missing is a model for building/distributing those C libraries with a java application.
Every ffi example I've found seem to operate on the assumption that you want to invoke syscalls or libc, which (with possibly the exception of like madvise and aioring) Java already mostly has decent facilities to interact with even without native calls.
> Every ffi example I've found seem to operate on the assumption that you want to invoke syscalls or libc ... Java already mostly has decent facilities to interact with even without native calls.
Because you would use ffi to interact with libraries that don't have Java wrappers yet: IE, you're writing the wrapper.
Using syscalls or libc is a way to write an example against a known library that you're probably familiar with.
The recommended distribution model for Java applications is a jlinked runtime image [1], which supports including native libraries in the image.
[1]: Technically, this is the only distribution model because all Java runtimes as of JDK 9 are created with jlink, including the runtime included in the JDK (which many people use as-is), but I mean a custom runtime packaged with the application.
Absolutely not. jlink is used to distribute applications (it includes your code, the Java libs you use, i.e. their jars, and the trimmed-down JVM with the modules you're using so that your distribution is not so big - typically around 30MB).
Java libraries are still obtained from Maven repositories via Maven/Gradle/Ant/Bazel/etc.
If you distribute libraries as jmod files, which few libraries do (in that case, jlink would automatically extract the native libraries and place them in the appropriate location).
So, other people have already answered this, but this does seem to be a gap where many developers lack some piece of knowledge to chain the whole solution together. You normally package this sort of thing by putting the native library in a jar, extracting it to a tmp file that will be deleted on exit, and opening that dynamic library.
I’ve met many perfectly reasonable developers who do know all those steps can be done but can’t put them all together - maybe because it just hasn’t clicked that you can store a library in a jar. It feels like something tutorials should cover, but I think falls into the, “surely everyone can work it out?” category.
Technically memfd_create will let you create a file descriptor backed by a memory region. However in Linux I don't believe there's a way to dlopen that. (Maybe dlopen /dev/fd/... might work?) In FreeBSD there's a fdlopen library function.
If your app is open source, or you're willing to buy a commercial tool, then you could try Conveyor from my company [1]. It will:
- Find all the shared libraries in your JARs or configured app inputs (files in your build/source tree)
- Sniff them to figure out what OS and CPU arch they are for
- Bundle them into the right package for each platform that it makes, in the right place to be found by System.loadLibrary()
- Sign them if necessary
- Delete them from the JARs now they are extracted. Optionally extract them from library JARs, sign them and then put them back if your library refuses to load the shared library from disk instead of unpacking it (most libs don't need this)
- JLink a bundled JVM for your app for each platform you target, using jdeps to figure out the right set of modules, and combine that with your shared libs.
When building Debian/Ubuntu packages it will also:
- Read the .so library dependencies, look up the packages that contain those other shared libraries and add package dependencies on those packages, so "apt install" will do the right thing.
So that makes it a lot easier to distribute Java apps that use native code.
I was on the assumption that it was dynamically linking the libarary with the OS dynamic linker, which in no OS I'm aware of is capable of loading libraries inside of zip files.
Not sure where I got that notion. Maybe I was overthinking this.
Yes. Check out a library like zstd-jni. You'll find native libraries inside it. It'll load from the classpath first, and then ask the OS linker to find it.
I'd like to learn how they do it. Because last time I've looked at this, the suggested solution was to copy the binaries from claspath (eg: the jar) into a temporary folder then load it from there. It feels icky :)
Do NOT force the class loader to unload the native library, since
that introduces issues with cleaning up any extant JNA bits
(e.g. Memory) which may still need use of the library before shutdown.
Remove any automatically unpacked native library. Forcing the class
loader to unload it first is only required on Windows, since the
temporary native library is still "in use" and can't be deleted until
the native library is removed from its class loader. Any deferred
execution we might install at this point would prevent the Native
class and its class loader from being GC'd, so we instead force
the native library unload just a little bit prematurely.
Users reported occasional access violation errors during shutdown.
Ah, looking through the docs [1]; you have to use your own ClassLoader (so it can be garbage-collected), and statically-link with a JNI library which is unloaded when the ClassLoader is garbage-collected.
I have some extremely unwieldy off-heap operations currently implemented in Java (like quicksort for 128 bit records) that would be very nice to offload as FFI calls to the corresponding a single-line C++ function.
Because "some inconvenience/unmet requirement" from a language is not an invitation to "throw out the whole platform and your existing code and tooling, and learn/adopt/use an entirely different, single-vendor platform".
Except if we're talking about some college student or hobbyist picking their first language and exploring the language space...
Assuming it is "sort for 128bit records", that's something C# does really well - writing optimized code with structs / Vector128<T> / pointer arithmetic when really needed without going through FFI and having to maintain separate build step and project parts for a different platform.
But even if it was needed, such records can be commonly represented by the same structs both at C#'s and C++'s sides without overhead.
An array of such could be passed as is as a pointer, or vice versa - a buffer of struts allocated in C/C++ can be wrapped in a Span<Record128> and transparently interact with the rest of standard library without having to touch unsafe (aside from eventually freeing it, should that be necessary).
I remember using Sqlite Java and not having to install sqlite on the image. Then I looked inside the Sqlite-java's jar and they just packed the sqlite binaries for the different OSs in the jar!!
Not sure if this is still the case, but one of the Java Sqlite driver used something called NestedVM to run Sqlite in the JVM when a native library wasn't available. It worked by cross-compiling the code to mips, then transpiling the mips assembly to Java byte code. I can't remember if it bridged system calls or libc calls to Java for things like file IO.
I once (2016 ish) used a serial-port library for Java. Needed to be cross platform desktop app for Linux, Windows and Mac (in that order, all on x86/64). And it was. I have forgotten the name of the library project I included, but it included DLL binaries for the platforms we were targeting.
You can't be as aggressive at removing functions in Java than in Rust though since it's dynamic dispatch (e.g., if you use toString once in your code, you need to keep all implementations of toString which are reachable even if users don't use reflection).
Static linking in Go and Rust includes compiled code only for the target platform. It does not include compiled code for every possible architecture, including 32-bit MacOS and Solaris on PowerPC.
Every ffi example I've found seem to operate on the assumption that you want to invoke syscalls or libc, which (with possibly the exception of like madvise and aioring) Java already mostly has decent facilities to interact with even without native calls.