Early on (15+ years ago) I spent a few weeks there on contract and I noticed they used Java EVERYWHERE, and not always well. They had a CS app named after a key Star Wars character that was in all likelihood a breach of the Geneva Convention. A code atrocity with the performance of a sloth on its 8th bong rip with a UX from hell.
If it helps that CS app was rewritten about 10 years ago (when I worked there, but not on that app) in part due to the complaints you mention. It's totally true that most resources were spent on customer facing apps. Internal apps were definitely not of the same quality, because they didn't need to be.
Good to hear. The fact that in 2005 you had an app that required seemingly petabytes of memory to operate, and put on machines barely powerful enough to play minesweeper, was in and of itself a series of bad decisions...but the app itself, and it's layout were just maddening. It's like MC Escher was the UX lead.
For some reason whenever I'm on my work's VPN, Apple Music lets me play 1 album and then the next time I try to start a song it will tell me I'm not logged in and I'll have to force quit and relaunch (frequently a few times) before it will let me play another album.
Apple Music is the only app that has this problem.
When I was there a decade ago, it started becoming more polyglot friendly (node apps had to use a jvm sidecar to do internal communications originally!)
My team wrote some of the Python libraries for internal services just so we could avoid that Java sidecar! It took 10 times longer to boot the sidecar than the Python app.
Another reminder that acronyms are pretty terrible for communication. Every time I onboard with a new org there’s a whole new set of acronyms to learn that’s barely faster than typing out the unabbreviated version. Nice to save a couple seconds when the cost is only a bunch of people not able to follow along when people are communicating.
To be clear: not ragging on OP in particular at all but more at the widespread practice at a company level.
> Another reminder that acronyms are pretty terrible for communication. Every time I onboard with a new org there’s a whole new set of acronyms to learn that’s barely faster than typing out the unabbreviated version
My (cynical) take on the matter is that software engineers want to prove how smart they are to other software engineers, and they think using obscure jargon and abbreviations is a good way to do it.
I’m sure that’s some part of it. I think another part, kind of similar, is acronyms being used as a form of in-group/out-group behavior. “I know the lingo so I’m in the club. You don’t so you aren’t”
Interesting the article jumps straight from REST to GraphQL and forgets Falcor[0] - Netflix's alternative vision for federated services. For a while it looked like it might be a contender to GraphQL but it never really seemed to take off despite being simpler to adopt.
Falcor is actually part of the "old" architecture described in the talk. Because it's mostly unknown and no longer used I didn't go into the details of it.
Falcor was developed at the time Facebook was developing GraphQL in-house. It has similar concepts, but never took off the way GraphQL did.
I was at the React Rally conference where Falcon was publcly announced in August of 2015. I recall that Facebook gave a GraphQL presentation right before.
It seems GraphQL was first announced publicly in February 2015.
I think this is a 20% improved utilization of CPU, earlier app was memory-bound or/and GC was consuming CPU. Now app has 20% more CPU available. It should be doing correspondingly more work. This could definitely be written clearly.
> Bakker provided a retrospective of their JDK 17 upgrade that provided performance benefits, especially since they were running JDK 8 as recently as this year. Netflix observed a 20% increase of CPU usage
Seems like it's exactly that, OP cropped out the relevant bit where they list it having an overall performance benefit for that extra CPU time. Otherwise it could be assumed that it just hogs more CPU to get the same result.
I haven't dealt with this side of Java in a while, but it reflects my experience poking at Java 8 performance. At some (surprisingly early) point you'd hit a performance wall due to saturating the memory bus.
A new GC could alleviate this by either going easier on the memory itself, or by doing allocations in a way that achieves better locality of reference.
Most modern GCs trade off CPU usage and latency. Less latency means the CPU has to do more work on e.g. a separate thread to figure out what can be garbage collected. JDK 8 wouldn't have had the G1 collector (I think, or at least a really old version of that) and they would have probably been using one of the now deprecated garbage collectors that would be collecting less often but have a more open ended stop the world phase. It used to be that this would require careful tuning and could get out of hand and start taking seconds.
The new ZGC uses more CPU but it provides some hard guarantees that it won't block for more than a certain amount of milliseconds. And it supports much larger heap sizes. More CPU sounds worse than it is because you wouldn't want to run your application servers anywhere near 100% CPU typically anyway. So, there is a bit of wiggle room. Also, if your garbage collector is struggling, it's probably because you are nearly running out of memory. So, more memory is the solution in that case.
The figure is about the overall improvement, not sure why that reads increase.
On JDK 8 we are using G1 for our modern application stack, and we saw a reduction in CPU utilisation with the upgrade with few exceptions (saw what I believe is our first regression today: a busy wait in ForkJoinPool with parallel streams; fixed in 19 and later it seems).
G1 has seen the greatest improvement from 8 to 17 compared to its counterparts, and you also see reduced allocation rates due to compact strings (20-30%), so that reduces GC total time.
It's a virtuous cycle for the GRPC services doing the heavy lifting: reduced pauses means reduced tail latencies, fewer server cancellations and client hedging and retries. So improvements to application throughput reduce RPS, and further reduce required capacity over and above the CPU utilisation reduction due to efficiency improvements.
JDK 21 is a much more modest improvement upgrading from 17, perhaps 3%. Virtual threads are incredibly impressive work, and despite having an already highly asynchronous/non-blocking stack, expect to see many benefits. Generational ZGC is fantastic, but losing compressed oops (it requires 64-bit pointers) is about a 20% memory penalty. Haven't yet done a head to head with Genshen. We already have some JDK 21 in production, including a very large DGS service.
A somewhat common problem is to be limited by the throughput of CPU heavy tasks while the OS reports lower than expected CPU usage. A lot of companies/teams just kind of handwave it away as "hyperthreading is weird", and allocate more machines. Actual causes might be poor cache usage causing programs to wait on data to be loaded from memory, which depending on the CPU metrics you use, may not show as CPU busy time.
For companies at much smaller scale than netflix where employee time is relatively more costly than computer time, this might even be the right decision. So you might end up with 20 servers at 50% usage, but using 10 servers will take twice as long but still appear to be at 50% usage.
If the bottlenecks and overhead are reduced such that it's able to make more full use of the CPU, you might be able to reduce to e.g. 15 machines at 75% CPU usage. Consequently the increased CPU usage represents more efficient use of resources.
>> while the OS reports lower than expected CPU usage
>> which depending on the CPU metrics you use, may not show as CPU busy time
If your userspace process is waiting on memory (be that cache, or RAM) then you’ll show as CPU busy when you look in top or whatever - even though if you look under the covers such as via perf counters, you’ll see a lack of instructions executed.
The CPU is busy in this case and the OS won’t context switch to another task, your stalled process will be treated as running by the OS. At the hardware thread level then it will hopefully use the opportunity to run another thread thanks to hyper threading but at the OS level your process will show user space cpu bound. You’ll have to look at perf counters to see what’s actually happening.
>> you might end up with 20 servers at 50% usage, but using 10 servers will take twice as long but still appear to be at 50% usage.
Queue theory is fascinating, the latency change when dropping to half the servers may not be just a doubling. It depends on queue arrival rate and processing time but the results can be wild, like 10x worse.
When you put it like that, yes. Hardware is cheap and all that. In practice I think that an organization that doesn't understand the software it is developing has a people problem. And people problems generally can't be solved with hardware.
If somebody knows how to make that insight actionable, let me know. No, hiring new people is not the answer. In all likelihood that swaps one hard problem for an even harder.
IMHO, Usually the people problem is that there are too many people working on the same machine. Sometimes that's unavoidable.
Sometimes, honestly, understanding the software its developing isn't an important business goal. It makes me personally angry, but most businesses do right by not picking business goals to placate me.
Sometimes you just have too many people.
Sometimes you can restructure your software and systems so that fewer people are working on a system and they can understand it better. Sometimes that would also involve restructuring your organization, which has pluses and minuses.
If you can ensure the smaller teams run similar stacks, there can be some good knowledge transfer when one team figures out an underlying truth about the platform that could apply elsewhere. And sometimes you get a platform expert team that can help with understanding and problem solving throughout the teams.
> Help me here, why do GC improvements cause CPU increase?
In Java 8 (afaik) there were pretty much no generational or concurrent garbage collectors, so garbage collector would happen in a stop-the-world manner: all work gets put on a halt, garbage collection happens, then the work can resume.
If you have a better GC, you have shorter and less frequent needs to do a stop the world pause.
Hence the code can run on cpu for more time, getting you higher cpu usage.
Higher cpu usage is often actually good in situations like this: it means you're getting more work done with the same cpu/memory configuration.
You have two kind of concurrent GCs, the ones where the marking phase in concurrent with the application and the ones where the evacuation phase is also concurrent with the application.
G1 only does the marking concurrently, the evacuation is done in small pauses.
A decade ago, there was only one concurrent evacuation GC available in Java, C4 from Azul. Now, we have Shenandoah and ZGC.
I haven’t seen the specific profiling data, but it’s possible that the garbage collector is running a collection thread, concurrently with regular processing threads, and thereby preventing entire world synchronization points which would idle processor cores.
Higher CPU usage paradoxically means better performance. When I last did OPS we used to watch total CPU usage of all services and if it was not 100%, then we started to look for a bottleneck to fix.
No, it's like improving a form to minimize the need for follow-up questions to the customer, and now seeing your workers (the same you had before) processing 20% more forms instead of waiting for responses.
Most of the postings for backend positions at Netflix I've seen call out nodejs. Can I assume they do both? Is one legacy and the other newer stuff, or are they more complimentary?
Things are certainly more of a blend now than what's presented in this presentation, but the presenter is a big Java platform guy here. I would say ~70% of the services I interact with on a day to day basis are Java, another 20% in Node, and then the last 10% is a hodgepodge of Python, Go, and more esoteric stuff.
It varies from team to team; the "Studio" organization that supports creating Netflix content does lots of nodeJS due to the perception that it's faster to iterate on a UI and API together if they're both in the same language. On my team, we're very close to 50/50 due to managing a bunch of backend, business process type systems (Java), and a very complex UI (with a NodeJS backing service to provide a graphql query layer). Regardless, the tooling is really quite good, so interacting with a Node service is roughly identical to interacting with a Java service is roughly identical to interacting with anything else. We lean into code generation for clients pretty heavily, so graphQL is a good fit, but gRPC and Swagger are still used pretty frequently.
No, just no. Performance and debugging are just plain horrible. The spring team loves to force you into their automagic shit and this bean stuff is so annoying. You almost got no compile time safety in this stack. It's the bane of my existence. I'd like to know that a compiled program will run. That seems virtually impossible with java/spring boot.
I'm not sure what "no compile time safety in this stack" even means in the context of a strongly-typed compiled language.
If you are referring to the dependency injection container making use of reflection, then Spring Native graduated from experimental add-on to part of the core framework some years ago. You can now opt for Quarkus/Micronaut-style static build-time dependency injection, and even AOT compilation to Go-style native executables, if you're willing to trade off the flexibility that comes with avoiding reflection. For example, not being able to use any of the "@ConditionalOnXXX" annotations to make your DI more dynamic.
(Personally, I don't believe that those trade-offs are worth it in most cases. And I believe that all the Spring magic in the universe doesn't amount to 10% of what Python brings to the table in a minimal Django/Flask/FastAPI microservice. But the option is there if your use case truly calls for it.)
Honestly, I've never run into anyone who considers Spring to be "the bane of their existence", where the real issue wasn't simply that the bulk of their experience was in something else. Where they weren't thrown into someone else's project, and resent working with decisions made by other people, but don't want to either dig in and learn the tech or else search for a new job where they get to make the choices on a greenfield project.
I've never used any other webstack (and I've used several, in several languages) where it was so hard to figure out why different tests pollute each other, causing subtle failures depending on the order of test execution. Sure, you could say that it's all a matter of learning the right way, but there are just a terrible amount of footguns.
Debugging why something doesn't work in Spring can also be a nightmare.
And the reference point for "compile-time safe framework" is of course not Django, which is written in a dynamically typed language...
> I've never used any other webstack (and I've used several, in several languages) where it was so hard to figure out why different tests pollute each other, causing subtle failures depending on the order of test execution. Sure, you could say that it's all a matter of learning the right way, but there are just a terrible amount of footguns.
> Debugging why something doesn't work in Spring can also be a nightmare.
Making these points you can see if somebody actually used Spring in production ;)
How much time we've lost with caching in tests and other shennanigans. I've never had a stack causing more problems by just being how it is than with Spring Boot. Don't even get me started on the absolute clusterfuck of hibernate. I'd love to move on from anything from this stack and we're working on it! Absolute dumpster fire.
No idea yet. But the first step is to have totally isolated domain logic in plain Kotlin. That's a nice start. Next we'll look into Graalvm, but no high hopes that everything we need is compatible.
/edit: it's not meant to replace spring, I know what these things do. It's to remediate some of the performance issues and compile time safety, which using graalvm inevitably brings ...
> "...failures depending on the order of test execution."
Every time I've seen issues where the order of test execution matters, it's either been: (1) someone one writing integration tests and calling them unit tests, or (2) transaction management issues with H2 or some other embedded test database.
For #1, you're going to have a bad time with any stack. For #2, transaction management with Spring Data JPA is admittedly a tricky subject to learn. However, most of the time you can simply slap a "@Transactional" annotation on unit test methods that mutate the database, and that will cure what ails you.
> "Debugging why something doesn't work in Spring can also be a nightmare."
I don't know what could make Spring more challenging to troubleshoot than any other Java framework, other than diagnosing issues with complex dependency injection config. And that is often grossly overblown. Most of the time, you don't HAVE to use reflection-heavy tools like "@Profile" or "@ConditionalOnXXX". Keep it simple and there's really not much magic there. And IntelliJ or any other professional-grade IDE can help to manage the magic you do choose to employ.
> "the reference point for "compile-time safe framework" is of course not Django"
I'm simply saying that in any "X Stack Considered Harmful" discussion, eventually you have to put your cards on the table and disclose which stack you are comparing X to.
Everything is written in Rust here in the imaginary fantasy world of HN. But back in the land of the real, almost all line-of-business server side API development is written in either Java/JVM, Python, .NET, Node, or rarely some PHP or Ruby holdovers. Roughly in that order. Occasional oddballs here or there using Go or something else, but not common enough to make a dent and typically very difficult to evangelise.
So with that palette available to me, I'm going paint server-side microservices with Java and Spring virtually every time. My default comparison is to Python simply because it seems the greatest rival today in terms of adoption. But if we're looking at strongly-typed compiled languages only, in the business world in practice that limits you to Java/JVM or C#/CLR. Which is like saying that Coke is the bane of your existence and your company should be drinking Pepsi.
> Every time I've seen issues where the order of test execution matters, it's either been: (1) someone one writing integration tests and calling them unit tests, or (2) transaction management issues with H2 or some other embedded test database.
Yeah, h2 is a very very bad idea. Hibernate and jpa are also garbage (proxies, lazy loading and transactions are just a big footgun). Have fun trying to use Sprint security and proxied hibernate objects. Somehow you have no guarantee that the class you receive and are checking and is guaranteed by code is what you expect it to be because of proxies. Absolute nightmare. We're using testcontainers, which has their issues on its own.
> I don't know what could make Spring more challenging to troubleshoot than any other Java framework
> "@Profile" or "@ConditionalOnXXX"
We have one configuration with @Profile to prevent it from configuring stuff in tests. Our biggest issue is that code that compiles is not guaranteed to run or even crash when running (@Lazy anyone?). I hate this. It caused us so much pain. A bigger issue really is the automagic I was quoting. They just configure shit together because they think it's nice. Also causes a lot of headache. You have to enable/disable a random bunch of shit until it somehow works. And after they released a version fixing vulns that were open for > 1 year because of old dependencies this vodoo breaks. (Especially the garbage that is spring security. Still no migration guide for ACl in v6 huh?)
Fun fact: not a fan of Rust. It's totally overblown for web backends. I love me some Go, but having the ability to use gradle multi-modules in a ports and adpaters architecture is just plain awesome. Not sure where the journey will take us, but Kotlin (which we use exclusively) is a lot of fun. I'd love to have native Kotlin at one point exclusively. No more failing builds because of a wrong jdk or anything.
Just everything about this ecosystem is fragile af. Not sure how we got to a point where shipping a simple Go bin is easier than shipping java. Don't get me started on the resource usage and performance.
> Every time I've seen issues where the order of test execution matters, it's either been [...]
Well, I've seen other cases - mostly regarded to mock pollution. And you may say that those are poorly written tests, and maybe you're right, but the problem - again - is that there are too many footguns. Even if you don't make mistakes, your coworkers probably will.
> I don't know what could make Spring more challenging to troubleshoot than any other Java framework
Here are some examples:
- Understanding what config and which beans are applied and why (especially in tests)
- Understanding and debugging @Transactional
- Understanding and debugging Spring Security
- Debugging why a request doesn't hit the controller method you think it should (this mostly relates to the former)
> I'm simply saying that in any "X Stack Considered Harmful" discussion, eventually you have to put your cards on the table and disclose which stack you are comparing X to.
Any framework that doesn't rely on reflection and prefers explicit config over implicit magic (that doesn't mean that there can't be any default values or behaviour). For the JVM stack, that could be something like Ktor, but there's others (even in pure Java). Even in dynamically typed languages, you have options like Sinatra and Flask. They can't give you type-safety, but at least they're more easily debuggable. Even Spring itself has tried to provide an alternative with Spring Fu, but unfortunately, there seems to be little momentum and it's still experimental: https://github.com/spring-projects-experimental/spring-fu
My experience, from memory; a few of the details might be wrong (e.g. it probably wasn't Redis, but it was some kind of datastore) but the basic experience is what happened:
Q: Why does my webapp run on most of our tomcats but not this one?
A: That tomcat server has the redis client libraries on the classpath, so Spring Boot automatically started up a connection pool connecting to localhost on the default port, and automatically includes that in the healthcheck endpoint. Since redis isn't actually running there, it fails to connect, so the healthcheck always fails and the app never shows as running, even though your code is working fine.
Q: How do I turn that off?
A: Upgrade to the next version of Spring boot and then add this undocumented annotation to your configuration
Q: How was I supposed to figure any of that out?
A: Hahaha fuck you
Spring proper is fine, useful even. Spring boot is an absolute nightmare of COME FROM style incomprehensibility.
Thanks for the example, I feel you. That's the essence of what I'm criticising. People go on about me being incompetent and not having read the documentation never seemed to have had this issues. I sometimes feel like even the spring devs don't understand what they're doing anymore.
I never hated Spring, I got used to working with it. I found the annotation-based approach to configuring classes really opaque though, and it was hard to understand how to configure things because those were also magic annotations, and then some things were beans and others weren't and... ugh.
I don't fault Spring for it, and we were on Java 11 back then with just a little bit of new hotness. Java itself just didn't lend itself to the best ergonomics.
And you could fault Ruby or any dynamic language for the same, but they usually save you a little bit of overhead or boilerplate.
I'm always interested in hearing people's alternatives. Modern C# and ASP.NET is likewise annotation based. Most contemporary Python frameworks heavily leverage decorators, which is just another name for annotations. Decorators have made their way into Typescript. Rust code is chock full of attributes, and I've even seen Golang libraries simulate annotations with the `reflect` package.
This never really sounds like a Spring or Java thing. It always sounds like a "not liking dependency-injection as a general pattern" thing. My issue with that is two-fold: (1) people should just say that, and not tie it specifically to any one stack, and (2) the alternatives seem to be either monkeypatching or else writing untestable code, and those alternatives are hardly any better.
I'm not saying annotations are bad though, I'm saying Spring's over-reliance on them makes the code harder to follow. Some of those annotations are actually done at compile-time via Lombok, which makes it even harder to know what the hell is happening.
Decorators and annotations are essentialy higher-order functions and higher-order classes, but there is nothing about them that makes it intuitive. They actually become quite difficult to write and much harder to debug, OOP's shit version of a monad.
Dependency injection is fine, but it doesn't need to be as heavy as it is in Spring. There are lightweight and compile-time safe dependency-injection frameworks, and it's not even that hard to write your own custom dependency-injection logic.
I was once dropped into a Spring codebase. I consider myself a decent programmer, but I couldn't navigate it at all. Other than that time I've been able to avoid magical frameworks in my career.
But I do want to learn Spring. What material would you recommend?
I like middle-ground dependency injection: do it, but only during build time.
Dagger library does that (generates all the glue code during pre-compile phase), so all the dependency-injection already happened before runtime. Also, JVM has an easier time reasoning/optimizing the code.
But I also found that writing the glue code by hand, same way as Dagger would do, is not that hard IMO.
> I'm not sure what "no compile time safety in this stack" even means in the context of a strongly-typed compiled language.
Easy: as I said down below, you can actually get wildly different classes because of things like Hibernate proxies. Also, you guessed it, the dependency injection part. I just hate it. Nothing works together. We have so many hacks and weird work-arounds because something doesn't work. (Websockets not working with tomcat for example, or the many funky troubles with using two modes of authentication at once).
> Honestly, I've never run into anyone who considers Spring to be "the bane of their existence", where the real issue wasn't simply that the bulk of their experience was in something else. Where they weren't thrown into someone else's project, and resent working with decisions made by other people, but don't want to either dig in and learn the tech or else search for a new job where they get to make the choices on a greenfield project.
In my previous startup, we used python and flask. Something I don't deem scalable for bigger teams/apps. We love Kotlin and Gradle (especially multi-modules). But there are so many drawbacks that just suck time. I have a bunch of private projects, all in Go. Fast and efficient as heck. Nothing I'd like to scale beyond maybe 5 people or 20k loc tho (no idea if bazel or something could help with that, no experience). You get a lot of good stuff, but you gain in fragility with Java/Kotlin.
Another point that just comes to mind is: how unsecure is this thing even? Dependencies are ages old, requiring you to litter your gradle build files with work arounds and overwrites so you're secure from some (often critica) vulns.
1. Hibernate isn't even part of Spring. You're free to use a newer ORM like jOOQ if you like, or Spring Data JDBC if you want to get closer to the raw SQL. This gets to what I was saying earlier... usually when I encounter someone with strong opinions on Spring, they haven't really dug in to learn much about it. They don't know where "Java" ends and "Spring" begins, they're just winging it and don't like that this doesn't work out well with larger team projects (arguably with ANY stack).
2. I don't care which programming language or framework you are using. If you hate dependency injection as a general pattern, then every alternative I've ever seen boils down to either: (1) monkeypatching all over the place to achieve the same goals, or else (2) just static hardcoding everything and not writing unit tests with any mocks. I mean, plenty of people utilize one of those approaches. They just usually don't do so while discussing safety and security with a straight face.
2. You cite Python and Go as alternatives, yet immediately acknowledge that they're unsuitable beyond small teams or apps (my God, I'll take dependency issues with Maven Central over PyPI any day of the week!). Honestly, this whole sub-thread seems to boil down to you just preferring to work on small codebases over large codebases. And that's perfectly fine! I just don't think that's language or framework-dependent.
You seem to read what you want to read. I was agreeing with you on the benefits of Java/Spring while still disliking it, citing my experience with go and python.
Your only argument is that if this stack makes problems, you just haven't read enough and must be an idiot winging it. The truth is that this whole thing, especially the standard spring stack including hibernate, is just one big footgun. You can't isolate any of these, as this stack intertwines to achieve this level of footgun concentration. Some faults are hibernate, some spring's, some is due to DI, and some are due to java. And I'm fed up with the level of trickery we need to have to work with this without issues. I'd like to focus on the bugs and shitty architecture I introduce instead of others.
> (1) monkeypatching all over the place to achieve the same goals, or else (2) just static hardcoding everything and not writing unit tests with any mocks. I mean, plenty of people utilize one of those approaches. They just usually don't do so while discussing safety and security with a straight face.
These are both stupid things to do in general. If you do proper hexagonal architecture you need neither of these hacks.
> "I was agreeing with you on the benefits of Java/Spring while still disliking it..."
Very well. You seemed to saying that Spring was far less attractive than some unnamed alternative, and I was trying to say that I don't see any such alternative in the same space. If you're basically saying, "Yeah, that's probably true, but I want to vent a little anyway", then fair enough.
I will however point out that Spring and Hibernate, etc are NOT tightly-coupled. You absolutely do not have to use any particular database persistence library with Spring, or any particular message bus framework or anything else. If you want to use jOOQ or MyBatis or raw JDBC or anything else, go nuts. The fact that most people gravitate toward certain de facto defaults doesn't make those requirements whatsoever.
> "If you do proper hexagonal architecture you need neither of these hacks."
I remember seeing that buzzword 15 or so years ago. From a glance at Wikipedia, this seems to be from the "Agile Manifesto" guy. It looks like most or all of this has been absorbed into 12-factor or microservices architecture or what have you. Regardless, I'd be surprised if it truly obviates the need for unit tests.
> Yeah, that's probably true, but I want to vent a little anyway
You got me!
> I will however point out that Spring and Hibernate, etc are NOT tightly-coupled. You absolutely do not have to use any particular database persistence library with Spring, or any particular message bus framework or anything else. If you want to use jOOQ or MyBatis or raw JDBC or anything else, go nuts. The fact that most people gravitate toward certain de facto defaults doesn't make those requirements whatsoever.
With tight coupling I actually meant tight integration and each bringing their own set of weirdness.
> Regardless, I'd be surprised if it truly obviates the need for unit tests.
Nah, absolutely not. But you get proper separation of concerns. You are almost doing it if you do spring. You just need a lot of interfaces and separate those concerns along specific lines. This gives you plain java domain code, isolated code for each type of side effects (be it web or files/database stuff) which you can test. You can also replace components very fast and easy without big refactorings.
My biggest pain with Spring/Hibernate/whatever is that it permeates every layer, even if you don't want it to. I brought up Hibernate because it took me ages, that Hibernate proxies everything and spring security doesn't understand that, doing funky things if you check proxied objects (don't get me started on string SpEL ... worst idea ever. Can't test shit and doesn't protect you from typos). And it just happened sometimes. That sometimes was when there was any lazy loading involved. Fun times!
With hexagonal (ports and adapters) you force yourself to separate the concerns. We also enforce it via gradle multi-modules and architecture tests. It works quite well so far. No damn compromises because hibernate forces you to have nullability in domain models where it wouldn't be possible from a domain point. It's all in different modules, hidden from everyone else. It also forces you to write domain code in plain java (or in our case kotlin). No funny side effects because of lazy loading, hibernate proxying, jackson funny business or anything else. It's been a bliss so far. We had to replace a lot of spring boot magic auto stuff (like anything hikari, hibernate, repos, models, DDLs, we need to do that manuall). But we can actually properly unit test code knowing that it can't fail a few layers up because of something missing. We can properly use Kotlin with all of its niftyness with sealed classes and so on. Really nice! Now we just need to solve the bean issues with additional tests and the typical gradle/java annoyances of throwing hurdles at you just because and then we have a very nice stack we love to use. And no footguns because we enforce a lot via compile time safety checks. We try to design everything so that it can't be used wrong. And best of all: a test suite without every test using something with transactions, a full application context or anything. Plain domain code, plain Kotlin tests. It's a big step up.
I can only recommend having a look into that. It's not that far away from a Spring Boot app with a lot of interfaces and Beans implementing that.
This sounds a bit negative but at the same time not purely related to Spring but rather to the choices of external dependencies. I've worked using Spring for years and it makes me smile reading it.
Hibernate looks easy but the abstractions have a cost associated with using and maintaining it. There are a lot of settings you need to get right. There were dedicated DBAs that would optimize it in the past. You could just use JPA to make the life simpler.
Tomcat.. I mean why? It was great but I'd say it maybe went out of pace compared to everything else. Why not embedded Jetty. At this point I'm starting to have doubts about how you deploy the services to begin with.
Dependency injection is actually great albeit the usual problem is that you need to read books to understand it as there is more than one way to do everything. My pain point was usually related to the differences you need to do among Java, Groovy, Kotlin but otherwise it is awesome.
Flask is shite, basic, Python has a hard time solving its mess with dependencies and its multithreading support is meh. Go is great I love it. But if you want to create enterprise software, Java ecosystem has you covered and the engineers are cheaper to hire.
Yeah, I'm also heavily ranting. My nick is meant as a hint haha. In the end and after quite a few years we're almost at a point where we love to work with our stack, minus the occasional weird funky thing happening because of the ecosystem (which would never happen in something like Go for example!). Hexagonal + Kotlin + Gradle multi-modules is a beast!
> My pain point was usually related to the differences you need to do among Java, Groovy, Kotlin but otherwise it is awesome.
Yup, agreed! We got to run kotlin pretty nice and got to play it together nicely with gradle convention plugins.
Agreed on Python. Was my first properly learned language back in the days and my first backend lang. Absolutely annoying. I also love go, but it lacks something like gradle, especially multi-module support. I love it but don't see it on a scale as our Kotlin codebase.
> engineers are cheaper to hire
I think there's also the benefit of an exotic language. You might only get 10 CVs, but you could probably hire half of them because only passionate people bother to look into exotic things. With java you have a bigger bandwidth ... which causes a lot of work.
I assume they mean problems with dependency injection.
If that's what they mean, I agree. I've seen dependency injection frameworks used in a bunch of different companies, and there are always people who consider it an essential lifesaver, like they just can't imagine working without it, and it always baffles me, because I've worked on equally large codebases that didn't use it, and it was occasionally a significant annoyance to pass dependencies by hand, but never equal to the annoyance of dealing with an automatic dependency injection framework.
This repeated experience of working with and without dependency injection, finding dependency injection to be at least as much hassle as it saves, and seeing that the people I've worked with who choose dependency injection have massively warped impressions of what it's like to work without it (they often think it's, like, impossible) has led me to see it as a tool that is driven mostly by FUD, at least at the scale of code that I have worked with.
And that's without even considering the deleterious effect that dependency injection has on design. In my cynical moments I think this is the real reason people love dependency injection. It's not that people hate the five minutes it takes to figure out how to manually pass a dependency to a module; it's that they hate the subconscious thinking that happens in those five minutes, as they see how the change affects the code, and it dawns on them that it's a code smell for every module to depend on everything else. Sometimes a dependency is a code smell, and dependency injection means you barely get a whiff, so faint you can pretend it's just your imagination. Doing it by hand means you get a few minutes to bask in the stench. You can't pretend you didn't notice.
Getting people to care about modularity, coupling, and cohesiveness in an application with dependency injection is markedly harder, just like it's harder in a language with global variables, just like it's harder to get people to think about APIs and modularity in a monolith than in microservices. And for me that's the worst part of working on codebases with pervasive dependency injection! Dependency injection is a massive liberating force for people who want to work without thinking about design. Instead of thinking about it, they just add another spaghetti dependency and keep on going, and do the same thing tomorrow, and the next day, and the next day. It's impossible to stop them! Adding a dependency is so immediate, so easy, there's no moment where they have to stop and think, "Hmmm, why am I using a dependency in module A that was only ever used in module B before? I'll need to instantiate it at the application level instead. But that means I need to pull some internal logic out of B so it can be run before B exists. Should I really be doing this? I'd better think/ask about this before I do it." Instead they just add an annotation and see if the dependency injection framework can figure it out. That's the only kind of problem I can think of that dependency injection excels at solving: problems that should never have been solved in the first place.
> "...it was occasionally a significant annoyance to pass dependencies by hand..."
Your comments are confusing on whether you eschew a dependency injection framework, or eschew dependency injection as a pattern in general.
If it's the former, then you are certainly free to roll your own DI rather than leveraging a library such as Spring, Quarkus, Micronaut, Guice, Dagger, or any of the others out there. I strongly doubt that you'll implement something better than any of those. However, having written the code yourself you'll be more likely to understand it all, and that can be attractive for many people.
If it's the latter, then it's harder to take the position seriously. Look, the Java ecosystem certainly suffered from extreme OO over-engineering in the late-1990's and early-2000's, no doubt about it. A lot of people went too far overboard with enterprise-y design patterns, for sure. But that pendulum has been swinging back for over a decade now... and in the 2020's, the point of using DI is not to go crazy with the enterprise design patterns. The point is write code that can support an actual unit test suite!
You use DI so that your test suites can inject mocks or stubs, to isolate external integrations from the code under test. I don't care which language or framework you use, or whether you're using a 3rd-party DI library or implementing the approach manually yourself. If you're not using DI to accomplish this in your tests, then you're either: (1) using monkeypatching to achieve the same goal in a more dangerous manner, or (2) just writing un-testable code, and perhaps writing some integration tests that you pretend are unit tests.
I think this is an example of what GP was talking about. DI is a way to manage the programming style, very common in Java, which relies on a huge amount of global mutable state. This makes testing and reasoning about code really hard. The alternative is a style which doesn't use so many global mutable variables. Whether you think of this as "not doing dependency injection" or as "doing dependency injection manually" probably depends on your personal programming journey.
I think you're blurring some lines when you present a choice between using a DI framework, rolling your own framework, or not using "dependency injection as a pattern in general." "Dependency injection as a pattern in general" is unavoidable in complex software and undoubtedly a necessary thing in many contexts, whereas dependency injection frameworks are tools that are not necessary for using "dependency injection as a pattern."
My preferred approach to dependency injection is using the plain mechanisms of the language I'm using to pass dependencies as constructor and function parameters, without any framework.
I pass test fakes the same way. When doing things this way feels burdensome due to overly complex method signatures or overly complex initialization code, I look for ways to improve my design.
This works at every scale of code I've worked at. It's possible there's some size of project that it doesn't scale to, but for every project I've worked on where another engineer swore that dependency injection was absolutely necessary on the project, I've worked on a larger project where nobody ever suggested dependency injection and everything was fine.
My experience doesn't prove that dependency injection isn't necessary at some larger scale that I haven't experienced, of course, but it does convince me that the industry is rife with programmers who think they can't get by without it at scales where it is not only unnecessary but probably harmful.
Spring is basically a standard in itself and it is easier to hire people in it. It also normalizes large pieces of the backend application so even though they are written by different people they are similar.
Once you learn the annotation based configuration it also saves a lot of time.
The performance is valid but it will only keep improving.
To be honest, over the almost 5 years we're into using it, I spent more time debugging funky stuff due to that design that it would've cost me to just do it by hand. By a big margin.
It's weird that some people including you directly attack my competence. As a power user you should have plenty of experience getting something to work that is not properly document, does not work how the documentation promised it to, or has weird problems on top of it. Look at idiotic things like this:
Take any similar issue and you'll see a bunch of people who try to find a solution for them because they just aren't repeatable at all. The underlying issue is the auto configuration doing things you can't follow quite properly. It's like it wasn't mean to be understood. Issues like the one I linked above also show me that the spring dev crowd also doesn't understand the ecosystem anymore. The problem is complexity and automagic.
I can definitely relate to these sorts of problems and I have faced them many times.
Honestly I have gotten so use to reading the decompiled source of code in IntelliJ to determine how to enable / disable items it is second nature.
Given the expressiveness and power behind the system that is a trade off I am willing to take. To a certain extent I think it is a trade off of using a tool that has so many permutations.
It's funny to see this perspective! I used to work in a few companies locally who had adopted the early java-ee style for their applications and my experience is exactly the opposite. When going to spring I'm usually diagnosing issues on the application layer (ie: business issues, not framework issues), while on the java-ee applications I was often having to fix issues down at the custom persistence layer each company had, etc.. I see where you come from having looked at the "old" spring stack (non -boot), and I can see people getting mad over the configuration hell and how stuff is hidden behind xml.. Much like how java-ee is!
I completely agree with this. Spring was an absolute nightmare during the short period of time where I had the misfortune of using it. It also didn't help that the codebase was a monstrosity... classes following no design patterns and having 40k lines. But still...
Agreed! Have fun reading spring source code or documentation to find anything. An obscure stackoverflow answer from 10 years back will hold the answer though. Fun times
That has not been my experience on the inside - I spend most of my days working on a Spring Boot based service at Netflix and frankly it's one of the most effortless environments I've ever worked in. Granted, there's a lot of ecosystem support from the rest of the company, but things are very low effort, and generally very predictable. I can usually drop a breakpoint in a debugger in exactly the right spot and find a problem immediately.
The issue with Spring ecosystem is that people use it without knowing why or which problem it solves but because almost everyone is using it. And most of the time, they don't need Spring (maybe a company like Netflix did, but it didn't prove to be the right choice at the end)
It's not quite as good as compile-time or type-based guarantees, but IME configuration errors with Spring are almost always flagged up immediately on application startup. As long as you have at least one test that initializes the application context (I.e. @SpringBootTest) then this should be caught easily
If you read more of my responses you'll see that I've been a Spring user for quite some time and know how it works. I just dislike how it works. Do you have any argumentation or did you just want to call me ignorant?
Performance and debugging simple, and compile time safety is Javas core domain. I think you're over focusing on proxying or enhancement of beans, but if you look at a documentation for a reasonable amount of time there's really nothing to it.
100% agree, Java and Spring are a mess and there's no justifiable reason to use them in 2023 (and no, "that's what we've always used" isn't a good justification)
Like srsly even DropWizard is better than Spring lol, let alone other even simpler frameworks like Ktor which is built on a much improved language over Java
What do you propose as an alternative? Something like Micronaut trades more compile time for stricter checks and faster runtime. Do you use something like that?
http4k[0] is a lightweight but fully-featured HTTP toolkit written in pure Kotlin that enables the serving and consuming of HTTP services in a functional and consistent way.
http4k consists of a lightweight core library, http4k-core, providing a base HTTP implementation and Server/Client implementations based on the JDK classes. Further servers, clients, serverless, templating, websockets capabilities are then implemented in add-on modules. http4k apps can be simply mounted into a running Server, Serverless platform, or compiled to GraalVM and run as a super-lightweight binary.
Apart the from Kotlin StdLib, http4k-core module has ZERO dependencies and weighs in at ~1mb. Add-on modules only have dependencies required for specific implementation.
I implemented a microservice with micronaut starting 3,5 years back. The fragility of krush/exposed is not very fun tho. we introduce another hurdle because we only use Kotlin. I'm open for ideas though.
PS: we replaced/will replace all microservices with Go.
Absolutely! I love how fast it is and how easy it is to write stuff real fast. Not a lot of footguns, except the pointer-for-loop thing afaik. It does feel clunky and quite manual sometimes!
Yes, either @IfBuildProfile/@DefaultBean if your logic depends on build profile, or @LookupIfProperty/@LookupUnlessProperty if it depends on property value
This is absolutely spot-on. The concepts that Johnson & co coalesced around, mainly IoC, DI, interfaces, and layers, are bedrock foundations for how most people write any service-oriented system. Say what you want about the things that Boot itself does, like auto configuration, or the Data persistence abstractions, but the foundations of the library are bedrock software engineering, so if you're gonna dump on something that's proven over 20+ years, you better come correct.
This "lingua franca" of Java development also allows you to hire, onboard, and get people productive - rapidly.
> It's plain this person lacks the experience and perspective to appreciate the problems a stack such as Spring solves
lol, having worked in Spring codebases for over a decade at this point, I assure you those of us who dislike Spring know exactly what problems it "solves" (which, btw, it doesn't)
if you can't seriously acknowledge and understand the mountains of criticism that has been leveled against Spring, by experienced and novices alike, you're the one who is arrogant and needs a reality check
but yeah it's typical of Spring fans to portray it as if Spring is The Only Professional Option when in reality there are much, much better, easier and simpler solutions for the same problems out there, including eg DropWizard and Ktor
Well that supports my point actually, that these "short comings" aren't short comings at all if one takes the time to learn the framework. Blabbering? People often say that who have no valid technical or intellectual response.
Spring is a safe and reliable choice I'd say; not the most exciting, but neither code nor frameworks should be exciting, they're used to solve a problem, they shouldn't become the problem itself.
GraphQL is interesting to me, I thought the clients were pretty similar across all platforms, meaning their API usage should also be similar enough to not need the flexible nature of GraphQL. But then, it allows for a lot more flexibility and decoupling - if a client needs an extra field, the API contract does not need to be updated, and not all clients need to be updated at once. Not all clients will be updated either, they will need to support 5-10+ year old clients that haven't updated yet for whichever reason.
Well, if the field is not available then new backend code will need to be written, resolvers, integrations, etc. But it does allow UIs to take less info over the wire, and eitherfewer joins need to be done or fewer performance-oriented APIs need building, as you say.
The stack is tremendously productive, but history has taught me a few things when dealing with Spring:
1. It's always best to start people off with plain old spring, even with an XML context, such that they understand the concepts at play with higher level abstractions like Boot. Hell, I even start with a servlet and singletons to elucidate the shortcomings of rolling your own.
2. Don't fall prey to hype around new projects in the Spring ecosystem, such as their OAuth2 implementation, since they often become abandonware. It's always best to take a wait and see approach
3. Spring Security is/was terrible to read, understand, and extend ;)
Ha ha, spring security is tricky and high chance may surprise some one while "boot"strapping a new project. But once done, it is out of way.
I did not like much of the XML, because it always seemed lot of duplication. All you doing is copying bean definitions and changing bean id and class/interface most of the time. But it became non issue over time. Now spring boot made it really easy with all those annotations.
I am a big fan of Spring Boot, its one of the few frameworks that just works and let me focus 100% on solving business problems. I've tried Micronaut, Quarkus, Dropwizard, but they slow me down too much compared to just using Spring Boot.
For me delivering business value is the most important metric when I am comparing frameworks. Spring Boot wins every time.
Netflix’s DGS framework for GraphQL is nice to work with but we’ve been frustrated with some prioritization choices by the team. For instance, if you’re using Kotlin, it’s impossible to define and pass scalars to the latest version of the client. There’s a year-old issue highlighting this problem that’s been ignored it seems.
This seems entirely unsurprising/standard Java setup. Perhaps it is proximity to Hollywood that some glamor is rubbed off on bog standard enterprise tech stack of Netflix.
IntelliJ is far and away above any other IDE and is well worth the paid license imho. It's a professional tool written by and for professionals. VSCode and Eclipse have some inertia in very particular workflows/tools and a few different (valid) ways of operating, but nothing is as polished and cohesive as the JetBrains.
Gradle, however, is a dumpster-fire of footguns and obtuse and non-debug-able DSLs.
gradle lets you get something custom working quickly, because it's basically a script with code you want to execute.
For small projects, it works fine, as long as the complexity doesn't grow beyond a certain point for the small project (aka, doesn't grow to a big project), and is maintained by the same person.
For a large project, i do not like gradle at all. Maven is a much better build tool, since standardization is the best thing since sliced bread.
Interesting. At one point it was fairly well "known" that Netflix used Scala. They presented at Scala conferences and had lots of open roles mentioning Scala. And Scala fans used Netflix as an example, claiming that they used Scala for recommendations, APIs, etc. But maybe it was all a psyop.
Not surprised about Rx. Rx is great at the UI layer imho, or anything with streams. For microservices, I don't see how it would have ever fit, since microservices should be as simple as possible doing just one thing.
When they say "applications" do they mean stuff with more meat than microservices? If they're mostly microservices, 2800 seems low to me for someone with the recognition factor of Netflix.
Ah, the way they break out artwork calls explains the weird behaviour I see with my U.K. Netflix account in Portugal - English titles, Portuguese posters, regardless of language preferences.
How does the artwork explain that? Wouldn't they just need to call the artwork service with your current language preference instead of the default language of your current geolocation?
Tell me ... why it's only netflix what makes CPU go on max? Why netflix is giving me info message when I turn on VPN in the middle of movie? Why it's only netflix who stop working another second when my internet connection has quick failure?
Sorry but it seems that you have no idea how netflix really works.
Do you even know what a frontend/client is vs a backend?
The frontend is anything but java (the website uses javascript, they have mobile apps, not sure what they are written in, but besides android having a few necessary wrappers, these are also not java applications).
You don’t see the backend and they do different things between different services so not much point in comparing them - that’s a huge, complex, partially cached-at-your-ISP even network infrastructure.
Besides, Java itself is very fast and is itself no reason for an applications’ (especially network-related) bad performance. It literally runs like half of the internet, with basically every other top 100 IT companies having business-critical infrastructure dependent on it.
The point that the others are trying to make is that these questions are not relevant in the context of the backend technology choices. The REST or GraphQL endpoints could be handled by monkeys writing bits via typewriters and it wouldn't affect the CPU usage in your browser.
Alright, its clear that either you’re a troll or you don’t realise that you have no idea what you’re talking about, either way there’s no point going further into this conversation
