This problem is not going to go away so easily. Numerous core Java classes (like BufferedInputStream) use synchronized. I count 1600+ usages in java.base. The blocking issue means it's _much_ easier to accidentally run into this, rather than waving it away as an unlikely edge case.
I personally ran into this Using the built in com.sun webserver, with a virtual thread executor. My VPS only has two CPUs which means the FJP that virtual threads run on only have 2 active threads at a time. I ran into this hang when some of the connection hung, blocking any further requests from being processed.
As the JEP states, pinning due to synchronized is a temporary issue. We didn't want to hold off releasing virtual threads until that matter is resolved (because users can resolve it themselves with additional work), but a fix already exists in the Loom repository, EA builds will be offered shortly for testing, and it will be delivered in a GA release soon.
Those who run into this issue and are unable or unwilling to do the work to avoid it (replacing synchronized with j.u.c locks) as explained in the adoption guide [1] may want to wait until the issue is resolved in the JDK.
I would strongly recommend that anyone adopting virtual threads read the adoption guide.
The problem is that it's rare to write code which uses no third-party libraries, and these third-party libraries (most written before Java virtual threads ever existed) have a good chance of using "synchronized" instead of other kinds of locks; and "synchronized" can be more robust than other kinds of locks (no risk of forgetting to release the lock, and on older JVMs, no risk of an out-of-memory while within the lock implementation breaking things), so people can prefer to use it whenever possible.
To me, this is a deal breaker; it makes it too risky to use virtual threads in most cases. It's better to wait for a newer Java LTS which can unmount virtual threads on "synchronized" blocks before starting to use it.
> have a good chance of using "synchronized" instead of other kinds of locks; and "synchronized" can be more robust than other kinds of locks (no risk of forgetting to release the lock, and on older JVMs, no risk of an out-of-memory while within the lock implementation breaking things),
I haven't professionally written Java in years, however from what I remember synchronized was considered evil from day one. You can't forget to release it, but you better got out of your way to allocate an internal object just for locking because you have no control who else might synchronize on your object and at that point you are only a bit of syntactic sugar away from a try { lock.lock();}finally{lock.unlock();} .
The fact that the monitor is public rarely causes issues, and in those cases where it's used on internal objects, it's not really public anyhow.
There's an additional benefit to using the built in monitors, and that has to do with heap allocation. The data structure for managing it is allocated lazily, only when contention is actually encountered. This means that "synchronized" can be used as a relatively low cost defensive coding practice in case an object which isn't intended to be used by multiple threads actually is.
Is there a similarly low-level synchronization mechanism that doesn't work this way? .NET's does the same thing.
I guess I might have preferred if both Java and .NET had chosen to use a dedicated mutex object instead of hanging the whole thing off of just any old instance of Object. But that would have its own downsides, and the designers might have good reason to decide that they were worse. Not being able to just reuse an existing object, for example, would increase heap allocations and the number of pointers to juggle, which might seriously limit the performance of multithreaded code that uses a very fine-grained locking scheme.
In .net async won where lock and mutex does not work (lock is like synchronized, not exactly the same, tough). That’s why most libraries use SemaphoreSlim which would work with green threads. But that’s more because of the ecosystem. I’ve barley stumble upon lock’s and mutex is mostly used in the main method since it acquires a real os mutex, not really a cheap thing but for GUIs it’s clever to check if the app is running. Most libs that use system.threading.task use semaphoreslim tough.
Yeah, definitely. But for a fair comparison I think you have to look at how .NET did things before async/await hit the scene. And, for that, the aspect of the design in question is quite similar between the two.
Early .Net is hardly an independent data point from early Java. Not only was .Net directly influenced by Java, it also had to support a direct migration from the Microsoft JVM specific Visual J++ to J#.
The handful of languages I know either do not have a top level object class that supports a randomized set of features ( C++ ) or prioritize a completely different way of concurrent execution ( Python, JavaScript ).
Hi Ron. Thanks a lot for the amazing work you are doing on loom and whole JVM platform. EA builds and GA release you mentioned can make it into 22 or you meant EA build for 23?
We make scalable graphics rendering servers to stream things like videogames across the web. When we started the project to switch to virtual threads we had that as number one on the big board. "Rewrite for reentrant locks."
Maybe we have more fastidious engineers than a normal company would since we are in the medical space? But even the juniors were reading and familiarizing themselves on how to properly lock in loom's infancy.
All that only to point out that, yes, they had communicated the proper use of reentrant locks long ago.
I do understand what you're saying from an engineering management perspective though. That effort cost a fortune. Especially when you have the FDA to deal with.
It was more than worth it though! In the world of cloud providers, efficiency is money.
We use the same technologies to deliver, say, remote CT review capability, that you would use to stream a videogame. It's just far more likely that the audience I'm communicating with, HN, is familiar with the requirements of videogame streaming, than it is that they are familiar with remote medical dataset viewing. Obviously the requirements or our use case are far more stringent, but no need to go into all that to illustrate the point made.
1 - Use virtual threads with reentrant locks if you need to do "true heavy" scaling.
2 - Kind of implied, but since you gave the opportunity to make it explicit with your comment =D, there is no need to waste your life on earning no money in videogames when the medical industry is right there willing to pay you 10x as much for the same skills. (Provided your skill is in the hard backend engine and physics work. They pay more for the ML too, if I'm being honest.)
In Virtual Threads: An Adoption Guide part there is:
When using virtual threads, if you want to limit the concurrency of accessing some service, you should use a construct designed specifically for that purpose: the Semaphore class.
That language only obliquely mentions the issue. It is nowhere near clear and direct enough for someone who is just, for example, using a third-party library that is affected. And then it's stuck inside detailed documentation that anyone who wasn't personally planning on adopting virtual threads is unlikely to read.
This seems like it's at least vaguely headed in the direction of that famous scene from early in The Hitchhiker's Guide to the Galaxy:
“But the plans were on display…”
“On display? I eventually had to go down to the cellar to find them.”
“That’s the display department.”
“With a flashlight.”
“Ah, well, the lights had probably gone.”
“So had the stairs.”
“But look, you found the notice, didn’t you?”
“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”
I would like to take this opportunity to thank pron and the amazing jdk developers for working on a state of the art runtime and language ecosystem and providing it for free. Please ignore the entitled, there are many many happy Dev's who can't thank you all enough.
People always forget that things that only happen every few million times, can happen fairly frequently on a busy server. This has bitten me numerous times. The nature of a lot of these types of issues is that they are hard to detect and hard to reproduce.
Virtual threads are nice for unblocking legacy code but they aren't without issues. There are better options for new code with less trade offs on the jvm as well. I've recently been experimenting with jasync-postgresql (there's a mysql variant as well) as an alternative to JDBC in Kotlin. It's a nice library. It does have some limitations and is a bit on the primitive side. But it appears to be somewhat widely used in various database frameworks for Scala, Java, and Kotlin.
Databases and database frameworks are an area on the JVM where there just is a huge amount of legacy code built on threads and blocking IO. It's probably one of the reasons Oracle worked on virtual threads as migrating away from these frameworks is unlikely to ever happen in a lot of code bases. So, waving a magic wand and making all that code non blocking is very attractive. But of course that magic has some hard limitations and synchronize blocks are one of those. I imagine they are working on improving that further.
> Virtual threads are nice for unblocking legacy code but they aren't without issues. There are better options for new code with less trade offs on the jvm as well.
The designers of Project Loom would say the exact opposite. The whole push behind Project Loom and similar models (Go's oft-praised "goroutines" runtimes being another one) is motivated by Threads being a much better fit for async behavior in a fundamentally procedural language like Java or Go than promise-based frameworks like async/await.
The whole motivation of Project Loom is to make the simple thing (spawning threads to handle blocking IO) the fast thing as well (by actually replacing the blocking IO with efficient async IO OS calls and managing the threads internally). Project Loom will be considered a full success if the next generation Java web server does something akin to "new Thread(() -> {executeHandlerFunc(conn); }.Start(); " for each incoming connection, just like the Go built-in web server.
I think it's not that black and white. Clearly they made a choice to be backwards compatible. Not because Java Threads have a nice API (not even close) but because a lot of legacy code that will never be changed uses it. Including all the ugly bits that you shouldn't be using. Like a lot of the low level synchronization primitives that date back to the early days of Java. It's an impressive bit of work but they made some compromises to make things work. A new API would have been easier, would have had less overhead, and be nicer to use. But backwards compatibility with legacy code was a big goal.
It mostly works fine and it's an impressive bit of engineering. But it has some really ugly failure modes in combination with hacky legacy code designed for real threads. So, you can't blindly assume things to just work. Hence the deadlocks.
Many Java servers already work the way you outline. It's just that they are a bit tedious to use with the traditional Java frameworks. Which is one reason I like using Spring's webflux with Kotlin instead. Just way nicer when it's all exposed via co-routines.
There are two separate choices. One is the choice of whether to implement green threads in the JVM at all, or whether to use async/await, or some other type of concurrency primitive. The other is whether to expose the new concurrency primitive using a new API or an existing one.
You could say the second choice, the specific API, was done, at least to some extent, for backwards compatibility reasons. I wouldn't agree, but I think there is at least some argument to be made. Here is one of the designer's explanation [0]:
> We also realized that implementing the existing thread API, so turning it into an abstraction with two different implementations won't add any runtime overhead. I also found that when talking about Java's new user mode threads back when this feature was in development, and back when we still called them fibers, every time I talked about them at conferences, I kept repeating myself and explaining that fibers are just like threads. After trying a few early access releases of the JDK with a fiber API, and then a thread API, we decided to go with the thread API.
However, the choice of adding a new concurrency primitive to Java in the form of green threads instead of others was very very clearly not done for backwards compatibility's sake. Ron Pressler (who is active here as 'pron') has several talks on the advantages of green threads over async/await that you can look at [0][1]. The designers of Go also had the same belief, and also chose to add green threads as the fundamental built-in concurrency primitive in Go, obviously not for backwards compatibility reasons in their case.
There might be some justification for comparing any one particular thing to the worst possible particular thing if those things have something in common. The only feature the two things you picked have in common is the word 'java'.
Appeal to expertise.
Appeal to authority is a falacy when the authority is not an expert in the requisite domain.
eg: we don't care what a policeman thinks about astrophysics, we do care what the astrophysicist says.
My understanding is that that highest performance webserver is nginx. And it uses async internally.
IMO, virtual threads is a better general purpose language feature because it avoids function coloring and is generally easier to reason about, but it may not result in the highest performance Java webserver.
NGINX is a native C implementation, so it has to be carefully written to use the OS's native high-performance IO and native OS threads.
The purpose of project Loom is to abstract that away from Java application code. The runtime can use the most efficient IO for the given platform (ideally io_uring on Linux or IOCP on Windows, for example) even if the application code calls the old blocking File.Write(). The application can then use simple APIs and code patterns, but still get massive performance.
With Loom, you can easily have 20,000 virtual threads servicing 20,000 concurrent HTTP requests and each "blocked" in IO, while only using, say, 100 OS threads that are polling an IOCP. A normal Linux box can typically only handle around maybe 1000 threads across all running processes.
Most application webservers (by default) handle one request per thread. For mostly IO bound stuff (which many projects are), it makes sense to me that threads become a bottleneck in relatively ordinary scenarios.
The scenario where your IO could handle way more than a thousand concurrent requests if only the thread overhead was reduced? When does that ever happen?
Each OS thread costs memory. With the version of Java I have, the default is to allocate 1MB of stack for each thread. So, 10,000 threads would require 10,000 MB of RAM even if we configured ulimit to allow that many threads. In contrast, asking the kernel to do buffered reads of 10,000 files in parallel requires much less memory - especially if most of those are actually the same physical file. Of course, they won't be read fully in parallel.
For example, this program:
var threads = new Thread[20000];
for (int i = 0; i < 20000; i++) {
threads[i] = Thread.ofVirtual().start(() -> {
try {
Files.copy(FileSystems.getDefault().getPath("abc.txt"), System.out);
} catch (IOException e) {
System.err.println("Error writing file");
e.printStackTrace();
}});
}
for (int i = 0; i < 20000; i++) {
threads[i].join();
}
Run as `java Test > ./cde.txt` takes about 4.5s to run on my WSL2 system with 2 cores, writing a 2 GB file (with abc.txt having 100KB); even this would be within the HTTP timeout, though users would certainly not be happy. Pretty sure a native Linux system on a machine beefy enough to be used as a web server would have no problem serving even larger files over a network like this.
1. You are not solving a real problem. The use case you describe (basically a CDN) is already exotic, the scenario where such a system would have already been implemented with Java and its basic IO seems implausible.
2. You did not compare against fewer threads to see if threads are actually the bottleneck rather than IO. Also, all your threads are competing for stdout.
The lack of support for synchronized isn't a fundamental or hard limit, it's just that the HotSpot implementation is complicated for performance reasons and they put off rewriting that code until later. They're indeed working on that now and in some future version I guess wait/notify and synchronized blocks will start to work. After all, you can easily transform such code into an equivalent that does work.
The system property jdk.tracePinnedThreads triggers a stack trace when a thread blocks while pinned. Running with -Djdk.tracePinnedThreads=full prints a complete stack trace when a thread blocks while pinned, highlighting native frames and frames holding monitors. Running with -Djdk.tracePinnedThreads=short limits the output to just the problematic frames.
Totally off topic but I am getting tired of the AI generated images used on nearly all blog posts nowadays. They are instantly recognisable, it just seems low effort and lowers the feeling of quality one might otherwise have
To me, its more about the style than the use of an AI. But I agree.
I enjoyed this writeup by Michael Lynch on finding an illustrator [1], for their blog. In doing some of my own writing, I've really found it enlightening how much secondary work goes into publishing your own work. I often think its so nice to be able to _just_ plug in what I want on a site and get a (more or less) free illustration. But as someone selling their own work / time, it feels wrong. I'd rather pay a real human and build a relationship and have something more quality. On the other hand, though, it can be expensive, time consuming, and I've been screwed over. Often it seems like a bigger risk than its worth.
So idk, you're trading some hardship and risk for an ethical dilemma but ease of use.
Between the shitty obviously-AI-generated square header image with floating hands everywhere, the equally shitty obviously-AI-generated image in the middle and the "Please pay for Medium" banner which takes up literally half the page, this blog post does its utmost to make a truly terrible impression.
There's a quality spectrum of AI-generated header images. Some are just random DALL-E output which aren't intrinsically relevant to the article (like the one used in this article), but you can have a little fun with it and do something distinct. This may require more control than just using Bing Image Creator.
Also, a thumbnail tip: square thumbnails are bad. If you have to use a square 1024x1024 AI generation, crop it to something like 1024x575, which incidentally can make things difficult if using AI generation since figuring out what to crop requires human intervention.
I just don't really understand the value they're supposed to bring.
If everyone uses the same looking generates images it just makes all the blogs look the same again.
Then the thing is, they usually have nothing to do with the actual article. So why not just leave them out and not waste space.
I dislike the style this particular author chose, but don't object in general. Assuming the images are actually somewhat relevant (or at least funny), I think I'd prefer an AI-generated image over a big wall of text.
The text is the only reason to visit the blog, so why waste bandwidth with images of four-handed people?
At least they could try and generate something where I can't see malformed bodies within seconds. Or create a nice diagram that actually adds something to the text.
Not really. I wish the trend of giant generic hero images on every blog post would go away, they almost never add any value. I think it was Medium that started the trend.
Unfortunately all social media sharing requires a thumbnail for easy clicking, no real way around it. (with Hacker News as the lone exception of course)
The default thumbnails in lieu of your own aren't good.
"Making clicks more likely" is a terrible measure of genuine value.
There are lots of images which will make people click, even if once they see your page they click 'Back' a second later. Our metrics are broken if we continue to attribute that click as 'success'.
> "Making clicks more likely" is a terrible measure of genuine value.
Genuine value, to who? For the author, getting more clicks is probably of "genuine value", depending on their goals for their writing. But seems most people are not writing and publishing stuff today because they think it provides value to others, but because they think it'll provide value to themselves somehow.
The question is: do you actually want to attract people who only click because of an image? And if you AI-generate it, are you fine with parts of the target audience not clicking on obvious AI thumbnails because they assume the entire content is low-effort?
One is aesthetic filler that is true to its purpose of loosening up the typography of a wall of text. The other tends to be awkwardly clever on a level of awkward that was unknown to mankind until recently. I used to hate stock photography fillers just like everybody else, but now my preference is as clear as it would be surprising to past me.
Interesting. Now that you mention it, there are illustrations there. But I'm pretty sure I subconsciously scrolled past them to get to the rest of the article without consciously noticing them on the first read.
Generated or hand drawn, they're kinda a wasted effort on a technical post.
At least you now know which of your peers have no taste and strange beauty standards. Some images posted by my colleagues for everyone to see on LinkedIn look like sexist propaganda cartoons.
Stock images used to hide this "quality" better than I thought.
Ah yes, let's take the most uncharitable explanation and assume that's the case.
Maybe they have no artistic ability of their own? Maybe they just aren't good at finding the kinds of images (that can be freely used without infringing on anyone's copyright) that they need?
If it were me, and the guidance was "never use AI generated images in your blog post", I would probably just not use any images at all. Which I guess for some people would probably be best. But personally I prefer walls of text to be broken up by... something.
That applies to every generative AI, not just images. Generate a bunch of text with LLMs and you'll also see patterns emerging that it won't ever break out of.
It is a known caveat that virtual threads do not work well with long running synchronization by pinning the thread. That unfortunately means that for many applications it may be premature to adopt them, but it is mature enough for broader evaluation by the libraries and frameworks. The Java team provided a status of their efforts recently [1].
Sorry, the first sentence is a mis-informing wording.
The `synchronized` pins the thread only when from within of the `synchronized` the program calls a blocking operation that would normally unmount the virtual thread, like blockingQueue.take() or similar. (Which is not a sane coding practice). It's because the unmounting, as it's implemented today, does not work well with synchronized.
It's better if people read JEP 444 than rely on forum comments, to avoid being misinformed.
Speaking of long-running - even without synchronized, a long running code keeps the native thread occupied, until some blocking operation is called. So an endless loop that does not call a virtual-thread-ready blocking operation will occupy the native thread forever.
Java virtual threads are a kind of cooperative multithreading - another virtual thread only gets chance to kick-in when some current virtual thread reaches specific blocking operations. In contrast to preemptive multi-threading with native threads.
So I agree with your conclusion. Virtual threads can not (yet?) be blindly used as a drop-in replacement of native threads for existing code. And the new code needs to take their specifics into account.
BTW, another method I discovered to block the native carrier thread that executes a virtual thread is to call blocking reading through FileInputStream, for example reading from the console. The FileInputStream does not implement virtual thread parking at all (yet?).
The issue in this case isn't actually the synchronized block. The thread is blocked on Object.wait, which releases the monitor before sleeping. The problem is that Object.wait is implemented in native code still, which pins the thread. The idea is that these days wait isn't exactly deprecated but there are better concurrency tools available, so they upgraded those first, leaving the Java 1 style concurrency tools for later. And Java 1 style concurrency has been improved on but is hardly insane, it can work well enough in many situations and is sometimes the basis for higher level concurrency utilities.
By long running I just meant anything that was not fast compute. I was more focused on finding the reference link so I agree my wording wasn’t clear.
Go started without preemption and added it later. The Java team has indicated a similar path, so we might see that tackled in the future. I think they could do that using safe points or JEP 312‘s handshakes, so it’s not infeasible.
For file io they wanted to explore io_ring and they might need to add a loom friendly resolver for JEP 418. There is just so much left, like scalable timers, that I think it’s going to be a long time until VTs will be a good default choice.
To get the whole context, so virtual threads are unusable?
What holds a monitor by default and is there a workaround?
Found more:
A virtual thread cannot be unmounted during blocking operations when it is pinned to its carrier. A virtual thread is pinned in the following situations:
The virtual thread runs code inside a synchronized block or method
The virtual thread runs a native method or a foreign function (see Foreign Function and Memory API)
For those those that don't know what this means: Blocking network TCP IO needs a sychronized block to work = you can't use virtual threads for networking. I wish they formulated it like that from the start!
Atleast now we know what they meant with don't use virtual threads for anything but tasks <- not blocking IO with synchronization!
So for now manual NIO is still the king of the hill.
We are reaching peak humanity levels of complexity!
> For those those that don't know what this means: Blocking network TCP IO needs a sychronized block to work = you can't use virtual threads for networking
That’s not true — blocking TCP IO is not implemented as blocking under the hood - that’s the whole point of virtual threads, so your conclusion is faulty.
Perhaps it’s better to say that they are not yet general purpose. There are many caveats which need to be resolved and are being actively worked on. I would not use them broadly yet, but that could change rapidly.
A monitor will pin the VT to the carrier thread. That can have surprising incompatibility in the current jdk. Soon these footguns will be fixed and you can use them worry free.
The problem is not “long running synchronization” but synchronization that relies on stuff running outside of virtual threads to unblock itself. There is no issue beyond performance if you perform filesystem operations in your mounted state.
Yeah, I wasn’t being particular about this exact issue and was generalizing about synchronization pinning the carrier. A deadlock is trivial once the implications of that are thought through.
Curious if you considered switching to a different connection pooling library. These days I usually use HikariCP which is fast an actively maintained. c3p0 hasn't had any activity for years, I'm not sure if it's still maintained.
Crucially, c3p0 will probably never see the `synchronized` blocks being replaced by reentrant locks. Since LTS offers exist for Java 21, many libraries might actually do that. But I actually hope that the ecosystem resists, which would force virtual thread users suffering from this problem to upgrade soon.
Perhaps we'll give HikariCP a chance. However, please keep in mind that the goal of the YDB team is to enhance database performance. We needed virtual threads to make TPC-C efficient enough to generate a reasonable load on a modest amount of hardware.
Virtual threads aren't necessarily faster, you still have just as many sockets and network connections as before. You can easily spawn 5000 platform threads, and if that's not enough, there are quite a few user-space implementations of fibers/coroutines/async etc on the JVM that can deal with many outlying requests (Cats/ZIO in Scala, Kotlin coroutines, the Play framework, concurrent.Future, etc.)
Been ages since I’ve touched it but back in 2017-2018 I had some fun integrating HikariCP in place of c3p0 in some Clojure projects and it was more performant.
Go has a mechanism to spawn a new thread (m in ho runtime parlance) if it thinks one of its threads might be blocked in a cgo (go’s “native function” equivalent). That prevents stuff like this.
Java does the same for Object.wait(), only the number of such compensating threads is limited by default, but can be extended via config option. They have exhausted the default number of compensating threads, I think.
And they are mistaken to call this situation a "pinning"
JEP 444:
> The vast majority of blocking operations in the JDK will unmount the virtual thread, freeing its carrier and the underlying OS thread to take on new work. However, some blocking operations in the JDK do not unmount the virtual thread, and thus block both its carrier and the underlying OS thread. This is because of limitations at either the OS level (e.g., many filesystem operations) or the JDK level (e.g., Object.wait()). The implementations of these blocking operations compensate for the capture of the OS thread by temporarily expanding the parallelism of the scheduler. Consequently, the number of platform threads in the scheduler's ForkJoinPool may temporarily exceed the number of available processors. The maximum number of platform threads available to the scheduler can be tuned with the system property jdk.virtualThreadScheduler.maxPoolSize.
(In my testing the default ForkJoinPool limit was 256)
So theoretically they could have extended the jdk.virtualThreadScheduler.maxPoolSize to a number sufficient for the use case. Although their workaround with semaphores is probably more reliable - no need to guess the sufficient number.
The situation with Object.wait() is not what JEP 444 calls "pinning". The "pinning" happens, for example, when one calls `syncronized(....) {blockingQueue.take()}`, which is not sane coding, BTW. In this case the native thread is blocked and is not compensated by another thread - much worse than the Object.wait(). The number of native threads that run virtual threads is equal to the number of CPUs by default, so "pinning" immediately makes one CPU unavailable to the virtual threads of the application.
All those issues are temporarily, as I understand. The JDK team works for fix Object.wait(), synchronized, etc.
> The situation with Object.wait() is not what JEP 444 calls "pinning". The "pinning" happens, for example, when one calls `syncronized(....) {blockingQueue.take()}` [...]
To call Object.wait() you need to own the objects monitor, which would imply that your code would actually look like `synchronized(....) {Object.wait()}` in which case you would indeed be pinned.
As I read JEP 444 (starting from the quote above and several following paragraphs, ending with the words "As always, strive to keep locking policies simple and clear."), the term "pinning" is when a blocking function, that normally unmounts virtual thread, does not do so, due to being called from `synchronized` or from native code.
That's different from blocking functions, described in the quote, that does not even try to unmount virtual thread. Like Object.wait().
Pinning is worse than those functions, because the functions compensate for a blocked native thread by adding one more native thread to the pool.
So does C# with active blocking detection (which injects threads to counteract this) and hill climbing algorithm to scale threadpool threads automatically.
It used to be the case - before .NET 6 there was only hill climbing so poorly written blocking code could starve threadpool very quickly (for + Task.Run + Thread.Sleep and the like), but since 6 blocking threads in such a way makes threadpool inject more threads without going through hill climbing mitigating the impact much more effectively. This does not mean such code should not be fixed however :)
> they chose to implement the scheduler in Java instead of C(++) and you're set for performance problems
The JDK has historically used some native implementations in its stdlib (zip, imageio and others), back when the runtime wasn't as fast as it is today. But today's runtime would often be faster in Java than those native implementations.
> Combine that with the fact that they chose to implement the scheduler in Java instead of C(++) and you're set for performance problems.
Ah yes, the argument from the 1990s. It would make sense to understand where the JVM and its compiler are these days before making incorrect statements about performance.
From your link:
> Blocking network TCP IO needs a sychronized block to work
This is a common problem when migrating a system from threads to virtual threads. In general, using primitives which block the current thread and prevent forward progress can quickly lead to deadlocks. It’s a hard issue to catch because in the past usually this would get “solved” by spawning a new thread to complete the task but in a world with virtual threads the runtime is usually reluctant to spawn more threads, so there’s nothing that can service more work if you’ve blocked all the threads.
Is that all that's happening here? There's an implicit limit on real threads, where before it was unlimited by virtue of not using the virtual thread's limited pool?
If it doesn't spawn threads when all of them are blocked, that seems kinda dumb. And a severe change in semantics. It can be conservative and try running unpinned ones on fewer threads and shuffle them around and slowly spawn more to ensure eventual progress, which would mean a possibly significant optimization problem, but a hard cap impacts correctness.
My long held belief: green/user-level/M:N threading schemes never work at first, and only work reliably after extreme effort has been put into fixing all the cases where blocking code gets called underneath. afaik there are only two modern working implementations: golang and erlang. This article is consistent with that belief.
There are many other implementations, although in less popular languages.
The trick is to include the green threads from the start, so there are no libraries that depend on real threading. That's why Go and Erlang are so successful.
It pins goroutine until it is explicitly released ensuring that multiple native calls will remain on the same platform thread and nothing else is going to use it.
This is critical for namespace manipulation on Linux.
Java only pins for duration of native call and synchronized blocks.
It looks like Java does not offer equivalent API?
For now could be achieved with synchronized but if synchronized will be changed in the future to not pin it would break.
Virtual threads were never intended as a drop-in replacement for platform threads. They offer the same API, but they are for different usage scenarios.
If you have lots of blocking I/O (meaning: waiting for things happening on other threads or processes, which offers scheduling opportunities), use virtual threads. If you compute or call native code, keep using platform threads.
The issue with synchronized is eventually going to be resolved. But long-running computations (sorting, parsing, number crunching, etc) or native calls must also in the future be offloaded to an ExecutorService with platform threads.
The change in semantics is that while in principle your OS thread will always have a turn at making progress (assuming no super heavy spin locks etc), that isn't true for virtual threads. The classic situation and the one they hit in the article is something like this,
You've got some virtual threads that encounter this code,
synchronized(foo) {
foo.wait()
}
And some other virtual threads that are in charge of awaking the waiters,
synchronized(foo) {
operation()
foo.notify()
}
This is a classic approach to the producer/consumer pattern in Java.
If operation() can do a virtual thread suspend, then it's possible to be suspended, relinquish the platform thread, which the scheduler reuses for the consumer and gets blocked on Object.wait. If this happens enough, you can end up with all the platform threads blocked, and no threads available to make progress on the producer.
The problem is that Object.wait doesn't release the virtual thread, which is a pretty major foot gun that I think the JDK team would have liked to avoid but it was too hard to implement correctly in the current JDK's codebase.
The only way I can see this being a problem is if the virtual threads can't be stolen from their (now pinned) carrier thread. Because otherwise that's all true of real threads too, blocking them is the whole point of Object.wait.
If there's no work-stealing from pinned carriers (or they're low-finite and normal threads are effectively infinite): yes that'd be a HUGE issue. I would be shocked if they released anything with that limitation though, that would violate some of the core expectations of mutexes and threads - independent ones need to make progress or nearly all patterns can't guarantee progress.
From Java docs for `jdk.virtualThreadScheduler.maxPoolSize`: the default is 256.
So yeah I can see that starving rather quickly, particularly with benchmarking-like workloads. Synchronized is very very common, 256 concurrent calls really doesn't seem all that abnormal.
If that were raised to like max-int32 would things be fine, semantically? That'd mimic real threads limits (no jvm limit at all afaict).
> If there's no work-stealing from pinned carriers (or they're low-finite and normal threads are effectively infinite): yes that'd be a HUGE issue. I would be shocked if they released anything with that limitation though, that would violate some of the core expectations of mutexes and threads - independent ones need to make progress or nearly all patterns can't guarantee progress.
Correct you can't steal the carrier thread from an Object.wait() waiting virtual thread. This is apparently in the pipeline but it is a pretty major limitation.
Most cases of synchronized/notify/wait should probably use concurrent collections instead (as message queues) so in greenfield code it's not that big of a deal. Virtual threads make writing consumers/producers using collections way easier too.
Sadly, most Java projects are not greenfield projects.
>Correct you can't steal the carrier thread from an Object.wait() waiting virtual thread. This is apparently in the pipeline but it is a pretty major limitation.
I mean stealing other virtual threads from the pinned carrier thread (except for the one pinning it) so they can make progress. Normal work-stealing stuff - the queue(thread) is blocked(pinned), so process that task(virtual thread) in a different queue(thread).
It makes sense that a pinned thread remains pinned with the virtual thread that pinned it.
The 256 default carrier thread limit is going to frequently be a problem though, yeah. That's more than enough to cause all this, and it's a pretty crazy default imo.
The `synchronized` by itself does not cause any problems for virtual threads.
Only when one calls a blocking operation from synchronized, the thread is not unmounted. E.g. `synchronized (...) {blockingQueue.take()}`. Note that this is not a sane coding practice. (Calling a potentially long operation from within synchronized. The blockingQueue.take() does not need to be wrapped into synchronized. It has synchronization inside and plays well with virtual threads. Only when wrapped into the synchronized, the current implementation can not unmount the virtual thread.).
The JDK team works to remove quirks like pinning in the future versions.
No, synchronized is a very primitive lock implementation compared to what's available in java.util.concurrent.Locks.
However, it's built directly into the JVM specification, so it's difficult to change while keeping compatibility, while j.u.c.Locks is just a library. In other words, they can't change synchronized schematic, so they created j.u.c.Locks as a replacement.
> it's difficult to change while keeping compatibility
Actually, it is trivial to change. Just embed a ReentrantLock into every object and rewrite all calls to "synchronized"/"Object.wait" to use that lock.
Unfortunately, this would result in a bit of performance regression (increasing per-object memory footprint). To solve that would require turning ReentrantLock into a magical intrinsic, fully integrated with lock bytes in the object header. Which is actually not that hard either — other runtimes like Golang or Android VM solve problems like this on daily basis. Oracle, however…
... was taking years to land Project Loom. So long that people started calling it vaporware. Project Valhalla is still regarded as such by many. It had to be shipped as soon as it was usable, even though a few rough edges remain that really ought to be deburred.
As you indicate, the complexity lies in not burning too many bridges with existing users and use cases. This is something that Android regularly does and which Go never really had to do due to its shorter history and up-front design.
Since Java 1.5 in 2004. Twenty years this year. Before that it existed as a separate library developed by Doug Lee. I remember using that library before Java 1.5 was released. There is quite a bit of Java code out there that predates that of course. Also, lots of people continued to not grasp the essentials of that library and stuck with the primitives they knew. So there's a lot of code there with synchronized blocks post Java 1.5 instead of the more robust concurrency primitives that came with the java.util.concurrent package.
Seems to be a similiar problem field as writing blocking functions that call async functions in C# and co-existence of synchronous and asynchronous code.
I thought the blog was great but "in summary" conclusion bad.
The summary merely stated that Java virtual thread are great. I expected a summary of the problem and solution, for example something like:
When using Java 21 virtual threads, you can end-up starved of carrier threads due to all carrier threads waiting on a pool exhausted resources with no thread available to free such resources. The solution is to wrap those resources in a virtual-thread aware object. In our case, we solved our problem by wrapping connections in semaphores.
The first issue is your second code is not Java (no await/async literal for Java yet)
The second issue is they're not completely equivalent. In the second case, you'd need extra memory for the `reentrantLock`, while `synchronized` works with any object. Furthermore, if you need to use `wait/notify`, then there need to be an extra `Condition` object to use in combination with the `ReentrantLock`. For sure, developers can rewrite most `synchronized` to use `ReentrantLock` and `Condition`, but javac won't do it automatically for you.
Why was c3p0 used (its latest version was released in Dec 2019)? Those tests existed for a while and people were too lazy to replace c3p0 with something newer? I guess that they spent all their time to use virtual threads in those tests and had no time left to look at c3p0.
I suppose that a hard limit on the number of carrier threads is a sensible choice then - deadlock is better than creating threads until the system grinds to a halt.
But then again, why couldn't scheduler detect a deadlock? Go has a system in place that, in case of total program deadlock, prints out an error message with all goroutines' stack traces, and stops the program. Perhaps Virtual Thread Scheduler could do the same thing?
But then again, Java also allows for native threads to run in parallel to Virtual Threads, which makes it impossible to detect whether there's a deadlock, and not just virtual threads waiting on a native thread.
I suppose this is a very good example why simple is better than complex.
Sounds similar to the quirks you get with TPL in .NET under some circumstances. For library code, a ConfigureAwait(false) invoke should be considered to signify that the execution does not need to resume on the original thread.
Why two sentences? Maybe you should ask ChatGPT if you want explanations with specific length requirements.
Anyway it's pretty simple really. A generic thread is bunch of stack frames (with their associated local variables). A standard OS thread is under the control of the kernel scheduler which decides whether the thread runs and makes progress or not. The VirtualThread in Java is just a thread which is not directly mapped to the OS thread scheduler but exists as a user space object that can be scheduled by a (Java implemented) scheduler. It's basically just a call stack with its local variables, but one that only steps forward when an OS thread of the scheduler decides to step it.
Edit: why are some commenters on HN so literal minded anyway? This is free form chat not code specs. You could have read "2 sentences" as "concise", you know...
Probably the author thought that this typical AI-generation quirks were a funny wink to the concept of virtual threads. The elephants, of course, represent Postgres.
Submitted title was "Java virtual threads caused a deadlock in TPC-C for PostgreSQL". We've reverted it now to the article's own title (truncated to fit HN's 80 char limit).
I would like to reply with a quote of sir Tony Hoare's 1980 ACM Turing Award Lecture: "There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies".
This one was a very large caveat pointed out loudly every time the feature was mentioned to the community. So, yes the limitation/flaw was very well known and is ideally going to be addressed in a future JDK release.
> This one was a very large caveat pointed out loudly every time the feature was mentioned to the community.
It really wasn't. There were people on here, including Oracle employees, claiming that the virtual thread implementation was a drop-in replacement that would work (not necessarily perform better, but work) in all cases.
And it indeed does in common usage scenarios. And also in this case once the issue with `synchronized` is resolved. After all, this is a benchmark and it's not surprising that one of the limitations of the design was hit.
I don't know if those Oracle employees actually did outright say -- or even imply -- "in all cases" as the GP asserted, but if they did, then "only" working in "common usage scenarios" would definitely be overselling the feature.
I find it unlikely as well that they said it would just work in all cases. But since it's going to work out eventually and a workaround exists, they would actually not be that wrong with that statement.
If your program requires 5 OS threads to be able to make progress (say, there are operations for which 4 threads are blocked while the 5th does work), and the runtime only spawns 2 OS threads and tries to schedule your 5 now virtual threads on those 2 OS threads, then your old program was perfectly correct and the virtual threads runtime has broken it.
This was a known and advertised failure case with the new threads runtime, exacerbated by limitations in the implementation that cause certain blocking operations to block the current OS thread instead of blocking the virtual thread and allowing another virtual thread to be re-use the underlying OS thread.
Having X resources and Y dedicated threads that operate on them, where Y > X, and allowing a thread to block in a way that requires the assistance of another thread to make progress but only when it's holding a resource, is a perfectly reasonable, standard, and safe design. When a change to the runtime silently reduces the number of synchronized sections a program can enter concurrently, it's not at all surprising that this breaks working code.
Article won’t load for me. My understanding was that the switch to virtual was supposed to be relatively simple, allowing the programmer to be naive. But a user deadlock is a user deadlock, no matter the threading impl.
The issue here is currently virtual threads don't work well with the 'sychronized' keyword. Right now synchronized will pin the carrier thread. The fix was to switch to a higher-level abstraction that works with virtual threads.
My understanding is there is work to make synchronized not pin the carrier thread, but that's some pretty complex and important code to change.
From a relatively brief skim and past Go and Java experience: synchronized blocks the current normal thread, so that doesn't really seem any different to me. If you starve your threads, you starve your threads.
It definitely leaves room to optimize by not pinning that thread, which would be great, but that shouldn't change semantics at all. Or is there something actually screwed up in the implementation of virtual threads that makes this a much bigger issue?
It’s a thread that supports n virtual threads. You want synchronized in virtual thread a and not the carrier thread which will block all the virtual threads.
Been away from Java land for a while. How did something like that even get into release? That’s like a pretty big loaded shotgun to leave lying around with lots of kids playing, no?
It's an explicitly documented shortcoming of the existing implementation that will be fixed soon. I knew immediately from the title of TA what probably happened. The other similar limitations (CPU-bound tasks, native calls) seem much more severe, but are ultimately unsolvable. Meanwhile, the issue with synchronized is regarded as a scalability bottleneck since the JDK is supposed to temporarily spawn additional platform threads. This behavior can be controlled via the system property `jdk.virtualThreadScheduler.maxPoolSize`.
Also, this is a benchmark. It's not surprising that they managed to produce a situation where more than n_cores virtual threads would actually start waiting.
Appreciate the reply. Hope you get it out soon since many /g do not read documentation and synchronized semantics changing is a ‘surprise’, specially since this one of those bugs that is a nursery for heisenbugs.
It spawns a new carrier thread in place, up to a certain, configurable limit. But starving carrier threads will also result in effectively live locks, so that’s not a solution.
So I don’t see the big fuss about it - don’t spawn a million virtual thread that all just spams synchronized?
Java's virtual threads are supposed to be a drop-in replacement for real threads. But using virtual threads means you get a far smaller number of real threads, and things that were safe back when you had an unlimited number of real threads available (or at least, a larger number than your database connection pool) are no longer safe.
Shouldn't you be able to use the same number of real threads though, plus some additional effectively-threads for the virtual threads that are not pinned? Doesn't seem like this should change semantics there, so the risk would be code that changes because of perceived advantages which are not true in edge cases - that's new behavior that wasn't possible before, there aren't really any existing semantics to break.
If they're, like, limiting to CPU cores * 2 threads: yeah that would be Bad™. Unambiguously. I haven't been able to find anything conclusive about this though.
Is there no limit (ignoring outside limits, e.g. from the OS) for normal threads? I know people usually use limited size thread pools for a variety of reasons, but I can't say that I've actually tried to exceed limits in a Java process yet...
That would indeed be a problem if it's not similarly unlimited by default. Configurable makes perfect sense, as does attempting to be conservative, but small hard-capped defaults are very obviously going to cause problems, especially while synchronized locks the carrier.
In this case the synchronized blocks are released by calls to Object.wait, so that code would not deadlock with normal threads.
The issue is that Object.wait doesn't suspend virtual threads, so you get deadlocks. The answer is to reimplement usages of the wait/notify pattern to use locks or concurrent collections (for example, using a concurrent message queue for the producer/consumer pattern, which is a common use case for synchronized/wait/notify).
This has nothing to do with postgres though. It’s part of a generic JDBC connection pool:
> The problem is that this synchronized code might be deeply embedded within the libraries you use. In our case, it was within the c3p0 library. So, the fix is straightforward: we simply wrapped the connection with a java.util.concurrent.Semaphore.
I bet if you just checked out connections and slept a random amount of time you’d have the same problem.
c3p0 appears to have been working under the assumption that their threads can always run when whatever they’re waiting on is done. Their only dependency is the OS giving them cycles.
Virtual threads changed the contract a little bit. Now one virtual thread running certain code can prevent a different virtual thread from ever getting any cycles even though they are not dependent on each other in the Java code. It’s a side effect of the current Java implementation.
The rules changed, and it tripped up c3p0. Unless they explicitly said somewhere that they were completely ready for virtual threads I’m not sure anyone is at fault here.
"Our PostgreSQL TPC-C implementation utilizes c3p0 for connection pooling...The problem is that...synchronized code might be deeply embedded within the libraries you use. In our case, it was within the c3p0 library. So, the fix is straightforward: we simply wrapped the connection with a java.util.concurrent.Semaphore. With this change, virtual threads are blocked on the semaphore and, crucially, release the carrier thread instead of delving inside c3p0. Thus, we never block inside c3p0 because we enter c3p0 code only when there is a free session available."
"Why not to use java virtual threads" -> Fundamental features of the language treated as implementation details in libraries for more than two decades cause deadlock.
"Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting."
Is this really that distracting to you that you cannot appreciate the content of the article? Just ignore it. Why are you in charge of how people express themselves on the internet.
We are in the goofy geo-cities days of AI. The art is strange and often illegal. The chatbots are vulnerable to "Grandma attacks". Enjoy the madness before the corpos kill the fun
I agree on an emotional level, but now I’m starting to wonder exactly why. Something similar from an illustrator would not be provoking to me.
Still working it out, but here’s a theory: when I’m sifting through stuff I’m looking for things that people spent time on. I have a pretty decent radar for effort, and effort is a good proxy for interesting. Even if I find it wrong or insightless, I still respect that people spend the time to express themselves.
However, these AI images are tricking my effort-heuristic and I’m annoyed that my attention has been temporarily hijacked someone spent 2 seconds on. This isn’t a new phenomenon, I’ve always been like this when I stumble across eg ads and low effort content marketing.
I think this cringe/cheesy feeling might be more a reaction to how it was made than the content itself.
I’ve come to see a lot of ai generated images as a proxy for effort, or care, and whenever I see them, my first thought is mostly “the author doesn’t care enough to make an effort about the presentation of their post”.
Same reason I stopped using GitHub copilot to write PR bodies, I had a teammate say “I know you didn’t put any effort or thought into writing it,so why should I bother reading it?” Which I resonate with.
I'm reminded of the way expensive restaurants use very simple descriptions, while less expensive ones load them up with superlatives and (uninformative) adjectives. The high end is confident enough that you assume it will be good.
The AI art assumes you need art to think that they've put in effort. More confidence would allow for simplicity.
But I cannot say if that confidence would be well placed. For all I know people really do think the art adds class.
Your comparison is indeed thoughtful. I would note a counterexample, though: an expensive restaurant with a very simple description gives no guarantee that you will be delighted, while some inexpensive places (possibly with basic ads) might offer something very delightful.
I would suggest looking at it from another perspective. Writing a text that precicely describes the image you want might require a lot of effort. Crafting this text to get as close as possible to the imagined picture also requires effort. Also, consider that some people can't draw (I also can't sing), but can write. AI gives a nice opportunity to add some color and visual humor, which is, in my humble opinion, great. But no doubt there are many examples of AI misusage (or overusage), which match your description.
I agree with your second point. But in this case, the AI art is a pure decoration.
Unless the author has some actual artistic talent, to me it feels like waste of time to spend more than a few minutes coming up with this. Some philosophers drinking Java coffee and playing with the PostgreSQL elefant. It doesn't even illustrate anything specific about the issue at hand.
If serious: that’s dismissive, superior, and a low-effort appeal to Haskell fans.
If not serious: it’s still low effort, but while it is framed as a zinger, it’s not funny at all. I don’t even understand what the humor might be, maybe it’s serious after all.
Switch to a language/runtime that's not only had virtual threads for decades, but also a saner synchronisation model (transactions) rather than synchronized blocks.
Exactly! It's the same thing with memory management primitives. If you expose malloc and free directly to an application programmer, things will eventually get buggy.
It's better to hide the locking primitives and let the runtime handle it for you safely.
Erlang is based on virtual threads (confusingly called processes).
The Erlang virtual machine schedules them on OS threads.
Erlang processes communicate using message passing, preventing deadlocks.
You can use millions of Erlang processes without problems, e.g., to handle millions of Elixir LiveView sessions.
Erlang has the advantage that it was built around processes and is effectively preemptive. Processes can be descheduled anytime they make a function call or use receive to get or wait for messages and being a functional language, you have a finite amount of instructions before calling a function.
Other languages adding virtual threads later in life don't have the same ability to feel preemptive. Although I think someone said Java has a nice trick or two?
Anyway, if all the virtual threads seem preemptive, you won't have the case that your limited number of actual threads are waiting on locks and not yielding --- all Erlang processes yield eventually; usually in a fairly short time frame.
You can have deadlock in Erlang, it's just a bit harder. It happens when two processes are both waiting on the other to send them a message which is analogous to two threads each waiting for a mutex the other holds. The same thing can happen in Go with its channels, another message passing based concurrency control mechanism.
Sure, you can make deadlocks in any language, but it's uncommon in Erlang.
Shared state is the exception, and message passing means that things that manage state, such as gen_servers, only process one message at a time from their inbox.
Contrast this with languages like Java where every object is a potential concurrency problem. Or the 10+ years of trying to make Python async (see Twisted).
I personally ran into this Using the built in com.sun webserver, with a virtual thread executor. My VPS only has two CPUs which means the FJP that virtual threads run on only have 2 active threads at a time. I ran into this hang when some of the connection hung, blocking any further requests from being processed.