(I don't know him personally, but have been following his blog for years.)
What these "just write SQL" rants are missing is encapsulation--let's say you've got a business logic, like "if the account is disabled, render the account name as 'Foo (disabled)'".
You want to write this logic in your preferred backend language, Go/TS/C/etc.
This works fine, in the /account/X endpoint (just load the account, and apply the logic).
But now what about the /accounts/client:Y endpoint (load all accounts, all the logic for all of their accounts)
As time goes by, you end up having 10-20 endpoints that all "return some part of account" as their payload, and you want the same "Foo (disabled)" business logic.
Your options are:
1. Build a single, giant SQL statement that strings together every snippet of business logic applicable to this endpoint (bulk friendly b/c the db is doing all the cross-entity work w/joins)
2. Push the business logic down into the db layer (simple for this, just string concate with an if, but what about anything that is a loop? doable in SQL but tedious)
3. Use your language's abstractions, like functions, to organize the business logic (what Brandur is attempting to do).
Nearly everyone wants to do 3, because 1 doesn't scale as your business logic becomes more & more sophisticated (copy/pasting it around every endpoint's single-giant SQL statements, or pushing it down into the db as views/stored procedures).
> because 1 doesn't scale as your business logic becomes more & more sophisticated (copy/pasting it around every endpoint's single-giant SQL statements, or pushing it down into the db as views/stored procedures).
I think the idea that doing business logic inside the DB is an anti-pattern is cargo culting. You can quite easily store schema definitions, stored procedures, etc. inside version control. You can document it just as you would any other code. You don’t have to see what precisely the call is doing at all times, provided you understand its signature.
Letting the DB do more than just be a dumb data store is a great idea, IMO, and one that is too often eschewed in favor of overly-complicated schemes.
I know putting business logic in the DB has been used very successfully, but it also has some large downsides.
It's harder to express some things as SQL and your team will be less familiar with SQL than your preferred language. SQL language tooling (code autocomplete, testing libs, etc) is also far less mature than tooling for most languages.
It's harder to debug when things go wrong. Where do you put a breakpoint?
It's likely your team ends up splitting your business logic between your app and your DB, and then you have to figure out which part is writing incorrect data when there's a problem.
For my apps, I'm trying to set up the database to reject incorrect information (with types and constraints and sometimes triggers), but have the app code do all the business logic and writing data.
Honestly, it's too soon to tell whether my apps will be successful with this approach. They haven't yet gone through years of real world usage and requirements changing. But so far it's gone well!
This is true, but then, it's also harder to write Rust than Python. There are tradeoffs which are made for every choice.
> and your team will be less familiar with SQL than your preferred language.
Also true, but given how SQL is practically everywhere and is not likely to go away any time soon, I strongly feel that nearly every dev could benefit from learning it.
> SQL language tooling (code autocomplete, testing libs, etc) is also far less mature than tooling for most languages.
Again, true. Personally I dislike autocomplete functions (they break my flow, needing to parse the suggestion and accepting it, rather than just typing from muscle memory), but I get that others like them. Re: testing, I have an unpopular opinion: this matters very little compared to most other languages. Declarative languages like SQL or Terraform generally do not produce surprises. If you get an unexpected output, it's probably because you have incorrect logic. There are some testing frameworks that exist for various flavors of SQL should you wish, but IMO as long as you have some rigorous E2E tests, and your dev/staging environments are accurate and representative of prod, you'll be fine.
> For my apps, I'm trying to set up the database to reject incorrect information (with types and constraints and sometimes triggers), but have the app code do all the business logic and writing data.
Hey, you're doing better than most! I've never understood people's unwillingness to use things like length constraints. Sure, your app should ideally catch basic issues so they never have to make it to the DB, but I'd rather have the guarantee. The last thing you want is for someone to find a way to inject the entirety of Moby Dick into a `name` field.
for 3, you could write a stored proc to handle the various situations, and call that stored proc appropriately, letting the DB engine optimize appending "(disabled)".
however, I do wish Go had a map function ala python map() as opposed to just verbosely writing more for loops or a complicated thread-safe goroutine. Seems almost strange that Google, who popularized the mapreduce model, doesn't want it in Go.
Stored procedures are also screeched at as being anti-patterns, somehow. I don’t get it; you can store them in version control just like anything else, and add comments where necessary in the code base indicating what the DB is doing.
This works even in adhoc loops, i.e. if you have a lifecycle hook** of "after an author changes, do x/y/z logic", and you update 100 authors, every SQL operation invoked by those ~100 individual hooks is auto-batched.
We've been running this in production for ~4 years at this point, and haven't had an N+1 since then (although we didn't initially support auto-batch find queries; that came later).
Of course kudos to dataloader.
*everything --> any queries our "find" API supports, which doesn't do aggregates, sums, havings, etc.
**lifecycle hooks --> yes, a blessing and a curse; we're always attempting to find better/higher-level abstractions for declaring the intent of business logic, than raw/imperative hooks.
Besides pitching Joist, going through OP, I'm not following how verbose the nested examples would get, i.e. ProductLoadBundle is loading "products & widgets".
But what if I need "products & widgets & widget orders". And like sometimes I want "just products & widgets" (2 levels) and sometimes I want "products & widgets & widget orders (3 levels)"?
Would these be the same "ProductLoadBundle" with some conditionals to the Product.LoadBundle method? Would the result of those conditionals be seen in the type-safe as, i.e. sometimes the 3rd level is available, sometimes it is not?
Or would it be two separate bundles, a 2-level Product.LoadTwoLevelBundle and a 3-level Product.LoadWidgetsAndOrdersBundle, which has the pro of better type-safety, but a con of repeating the bundle boilerplate for each unique shape/tree of data your app needs to load.
My guess/hope is that it's the 2nd, if only because I assume brandur also values type-safety over verbosity.
It took me awhile to find again, but this OG scala library in theory handled these adhoc shapes (called "enrichments" in their readme) somewhat elegantly:
Mea culpa another Joist pitch, but TypeScript makes this "give me an adhoc type based on the call-site specific tree of data" super-pleasant, i.e. Loaded<Product, "widgets"> vs. Loaded<Product, { widgets: "orders" }> gets you the two different 2-level vs. 3-level types.
...granted, you just have to accept non-Go/non-native levels of performance., but tradeoffs. :-)
I worked in a Rails app in ~2018 & hunting down N+1s by sprinkling `includes` in just the right places was tedious, despite being a regular occurrence.
Since then I've been building Joist, which is written in TypeScript, and has dataloader integrated into every operation (even `find`s) to basically never N+1:
Love a tiling WM post! Reading the "twos days to get setup" reminds me of fighting a blank xmonad setup back in the day :-), and makes me all the more appreciative of Regolith Desktop, a super-minimal Ubuntu+i3wm (and sway) setup; everything just works, Zoom sharing, multi-window, etc. Recommended!
And putting effort into adding every feature you need makes me appreciate it even more. I don't mean to say that humans derive value from suffering, but looking back at some of the headache and seeing that they are mostly solved now gives me this cool peace of mind. Plus, if anything breaks I know how to solve it. It is akin to moving from some batteries included distro like pop_os to arch. Nothing against pop, it is a wonderful effort, but reading through the documentation and putting your distro together gives you way more insight into the inner workings which is invaluable for when it breaks.
> reading through the documentation and putting your distro together gives you way more insight into the inner workings which is invaluable for when it breaks.
I definitely feel that way about the core libraries/frameworks I'm building my applications/codebases on top of, so can relate!
But for my WM/desktop, I'm pretty happy to a) not know it's innermost workings, and b) just have it not break in the first place! :-)
I guess we can each have our own yak-shaving preferences. :-)
I was exactly like that until kde decided it knew what was best for me and killed all the kwin scripts I relied upon with plasma 6 :(
But yeah! We can't know everything, it is very important to consciously decide what to be ignorant about. What is relevant to me might not be relevant to you and that is the beauty of it. If everyone got interested in the exact same thing, just imagine the chaos.
Wow, looks great! We currently happily use graphile-worker, and have two questions:
> full transactional enqueueing
Do you mean transactional within the same transaction as the application's own state?
My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality between steps within the Hatchet jobs...
I get that, but fwiw transactionality of "perform business logic against entities + job enqueue" (both for queuing the job itself, as well as work performed by workers) is the primary reason we're using a PG-based job queue, as then we avoid transactional outboxes for each queue/work step.
So, dunno, loosing that would be a big deal/kinda defeat the purpose (for us) of a PG-based queue.
2nd question, not to be a downer, but I'm just genuinely curious as a wanna-be dev infra/tooling engineer, but a) why take funding to build this (it seems bootstrappable? maybe that's naive), and b) why would YC keeping putting money into these "look really neat but ...surely?... will never be the 100x returns/billion dollar companies" dev infra startups? Or maybe I'm over-estimating the size of the return/exit necessary to make it worth their while.
> Do you mean transactional within the same transaction as the application's own state? My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality between steps within the Hatchet jobs...
Yeah, it's the latter, though we'd actually like to support the former in the longer term. There's no technical reason we can't write the workflow/task and read from the same table that you're enqueueing with in the same transaction as your application. That's the really exciting thing about the RiverQueue implementation, though it also illustrates how difficult it is to support every PG driver in an elegant way.
Transactional enqueueing is important for a whole bunch of other reasons though - like assigning workers, maintaining dependencies between tasks, implementing timeouts.
> why take funding to build this (it seems bootstrappable? maybe that's naive)
The thesis is that we can help some users offload their tasks infra with a hosted version, and hosted infra is hard to bootstrap.
> why would YC keeping putting money into these "look really neat but ...surely?... will never be the 100x returns/billion dollar companies" dev infra startups?
I think Cloudflare is an interesting example here. You could probably make similar arguments against a spam protection proxy, which was the initial service. But a lot of the core infrastructure needed for that service branches into a lot of other products, like a CDN or caching layer, or a more compelling, full-featured product like a WAF. I can't speak for YC or the other dev infra startups, but I imagine that's part of the thesis.
> a lot of the core infrastructure needed for that service branches
> into a lot of other products
Ah, I think I see what you mean--the goal isn't to be "just a job queue" in 2-5 years, it's to grow into a wider platform/ecosystem/etc.
Ngl I go back/forth between rooting for dev-founded VC companies like yourself, or Benjie, the guy behind graphile-worker, who is tip-toeing into being commercially-supported.
Like I want both to win (paydays all around! :-D), but the VC money just gives such a huge edge, of establishing a very high bar of polish / UX / docs / devrel / marketing, basically using loss-leading VC money for a bet that may/may not work out, that it's very hard for the independents to compete. I have honestly been hoping post-ZIRP would swing some advantage back to the Benjie's of the world, but looks like no/not yet.
...I say all of above ^ while myself working for a VC-backed prop-tech company...so, kinda the pot calling the kettle black. :-D
Good luck! The fairness/priority queues of Hatchet definitely seem novel, at least from what I'm used to, so will keep it bookmarked/in the tool chest.
Nice! We copied StyleX's "type-safe extensions" in Truss [1] so things like `<MyButton xss={Css.mt5.$} />` are allowed (setting margin is fine) while disallowing `<MyButton xss={Css.dg.$} />` (anything "not margin") that would mess up the components internal impl details with a compile error.
That said, we don't actually use the feature that much, vs. higher-level logical props like `<MyButton compact />`.
I know we're supposed to use build-time CSS-in-JS these days, but afaiu they don't support the rare-but-handy "just spread together ~4-5 different object literals from ~random different conditionals + props", i.e. intermixing styles some inside the component + outside the component, which emotion handles really well.
Basically this [2]. StyleX says it does "cross-file styles"...but can it support that? I kinda assume not, but I'm not sure.
StyleX is compiler-time only, so yes misses out on dynamic styles.
I made Tamagui a hybrid of compile and runtime, where the optimizing compiler actually handles object spreads, conditional logic, and even cross-module imports. It's really nice to get the near-0-runtime performance while maintaining all the benefits of dynamic styles.
I got the impression it was not dynamic values? Just that you can combine static objects dynamically. If I’m wrong I’ll re read the docs but someone else pointed that out to me and linked part of the docs that seemed to indicate that.
Cool, thanks Naman. It’s really great work overall, congrats on the launch. Maybe those couple places could have wording that indicates it supports dynamic as well.
Fwiw I appreciate your effort! That sounds really frustrating.
I agree the recent bun/tsx/esbuild (but bun especially) has shown the node CJS/ESM fiasco was a bit of an emperor-wearing-no-clothes moment, where I think us every-day JS programmers just trusted the node devs at their word, that CJS/ESM had to be this painful...
But now, seeing that it doesn't have to be that way, it's like wait a sec...the last ~5 years of pain could have been avoided with some more pragmatic choices? Oof.
Kinda surprised by all the "stuck on Yarn v1" comments so far...
We went through the same "~2015 everyone uses yarn", "~2020 everyone goes back to npm" cycle, but in the last ~year or so are back on yarn v3 for...reasons that I forget (oh yeah, the ability to download multiple platform libs for when you're running in docker [1])...but :shrug: it's been great.
My only complaint is that `yarn dedupe` should be automatic (as it was in yarn v1), or at least via a flag. :-D
But otherwise yarn v3 (with the node_modules linker) has been great, and look forward to using v4.
One of Bun's assertions is that their AsyncLocalStorage is significantly faster than Node's, b/c it purposefully only implements the subset of async_hooks that can be implemented w/o affecting performance:
(I don't know him personally, but have been following his blog for years.)
What these "just write SQL" rants are missing is encapsulation--let's say you've got a business logic, like "if the account is disabled, render the account name as 'Foo (disabled)'".
You want to write this logic in your preferred backend language, Go/TS/C/etc.
This works fine, in the /account/X endpoint (just load the account, and apply the logic).
But now what about the /accounts/client:Y endpoint (load all accounts, all the logic for all of their accounts)
As time goes by, you end up having 10-20 endpoints that all "return some part of account" as their payload, and you want the same "Foo (disabled)" business logic.
Your options are:
1. Build a single, giant SQL statement that strings together every snippet of business logic applicable to this endpoint (bulk friendly b/c the db is doing all the cross-entity work w/joins)
2. Push the business logic down into the db layer (simple for this, just string concate with an if, but what about anything that is a loop? doable in SQL but tedious)
3. Use your language's abstractions, like functions, to organize the business logic (what Brandur is attempting to do).
Nearly everyone wants to do 3, because 1 doesn't scale as your business logic becomes more & more sophisticated (copy/pasting it around every endpoint's single-giant SQL statements, or pushing it down into the db as views/stored procedures).