There's a balance with a DB. Doing 1 or 2 row queries 1000 times is obviously inefficient, but making a 1M row query can have it's own set of problems all the same (even if you need that 1M).
It'll depend on the hardware, but you really want to make sure that anything you do with a DB allows for other instances of your application a chance to also interact with the DB. Nothing worse than finding out the 2 row insert is being blocked by a million row read for 20 seconds.
There's also a question of when you should and shouldn't join data. It's not always a black and white "just let the DB handle it". Sometimes the better route to go down is to make 2 queries rather than joining, particularly if it's something where the main table pulls in 1000 rows with only 10 unique rows pulled from the subtable. Of course, this all depends on how wide these things are as well.
But 100% agree, ORMs are the worst way to handle all these things. They very rarely do the right thing out of the box and to make them fast you ultimately end up needing to comprehend the SQL they are emitting in the first place and potentially you end up writing custom SQL anyways.
The question is why we don’t have database management systems that integrate tightly with the progmming language. Instead we have to communicate between two different paradigms using a textual language, which is itself inefficient.
We tried that in 90’s RAD environments like Foxpro and others. If it fits the problem, they were great! If not, it’s even worse than with an ORM.
They rarely fit today since they were all (or mostly) local-first or even local-only. Scaling was either not possible or pretty difficult.
https://permazen.io/ exists and is a simpler yet still very powerful way to think about databases (for java but the concepts are general).
But it's only really efficient if it can run code right next to the data via fast access - ideally the same machine. The moment you have a DB running on separate hardware or far away from the client, it's going to be slower.
SQL is a very compact way to communicate what you want from a complex database in a way that can be statically analyzed and dynamically optimized. It's also sandboxable. Not so easily to replace.
> Because every single database vendor will try to lock down their users to their DBMS.
I mean, that already happens. It's quite rare to see someone migrate from one database to another. Even if they stuck to pure SQL for everything, it's still a pretty daunting process as Postgres SQL and MSSQL won't be the same thing.
> It's quite rare to see someone migrate from one database to another.
I'm not discounting the level of effort involved, but I think the reason you don't see this often is because it is rare that simply changing DBMS systems is beneficial in and of itself.
And even if it was frictionless (ie: if we had discovered ORM Samarkanda), the real choices are so limited that even if you did it regularly, you would soon run out of DBMSs to try.
It would obviously be beneficial to go from super expensive to free (Postgres). But no one does - why? Because sql is just a veneer over otherwise two completely different things.
I migrated a database with some stored procedures from MSSQL Server to Oracle.
Then lots of logic was added as stored procedures to the database.
Then I migrated the same system to MySQL. Including the SP.
Doesn't happen often, but it does happen.
The answer is simple: model optimized for storage and model designed for processing are two different things. The languages used to describe and query them have to be different.
There's a balance with a DB. Doing 1 or 2 row queries 1000 times is obviously inefficient, but making a 1M row query can have it's own set of problems all the same (even if you need that 1M).
It'll depend on the hardware, but you really want to make sure that anything you do with a DB allows for other instances of your application a chance to also interact with the DB. Nothing worse than finding out the 2 row insert is being blocked by a million row read for 20 seconds.
There's also a question of when you should and shouldn't join data. It's not always a black and white "just let the DB handle it". Sometimes the better route to go down is to make 2 queries rather than joining, particularly if it's something where the main table pulls in 1000 rows with only 10 unique rows pulled from the subtable. Of course, this all depends on how wide these things are as well.
But 100% agree, ORMs are the worst way to handle all these things. They very rarely do the right thing out of the box and to make them fast you ultimately end up needing to comprehend the SQL they are emitting in the first place and potentially you end up writing custom SQL anyways.