To me, this kind of 'magic' is definitely a double-edged sword. For the people that are familiar with the framework, I assume it makes it much nicer to write code and to reason over it (at the above-framework level), since there's no boilerplate to write or understand.
But as someone who often has to jump into unknown codebases and get up-to-speed quickly, I've certainly been confused by framework magic before, which makes it almost impossible to reason 'up' from the concrete implementation of e.g. a request handler. Since the keywords to grep for aren't explicitly mentioned in the handler anywhere and the 'dispatch' side of things is hidden under mountains of abstraction, you usually have to resort to tutorials to understand what the hell is going on, and even then the codebase might use some extra-magical properties of the framework that aren't clear to anyone that isn't at home within it.
So it could be argued that this magic saves you code in exchange for a need for more documentation, at which point you might just be better off being more explicit in your code -- at least explicit enough so things remain greppable.
> So it could be argued that this magic saves you code in exchange for a need for more documentation,
But not this:
> at which point you might just be better off being more explicit in your code
Reason being that you only need to add documentation once (for the framework), and then you can have cleaner more readable code for every project using it. Yes, there’s magic enabling it, but shouldn’t need to worry about how it works if it’s well documented enough. And if you really need to know then you can read a post like this one which explains it.
That is a fair point. I guess it basically comes down to preference:
Should code be essentially self-explanatory, without needing to resort to external documentation to understand, or can frameworks be terse to the point of being a kind of DSL?
I suspect most low-level programmers prefer the former ("It's all right there"), while dynamic-language coders feel comfortable with the latter ("I only want to be explicit where it's absolutely necessary.")
As another top-level comment pointed out, Rust is interesting because it attracts both kinds of programmers, so it's no wonder the question of which approach to go with for a Rust framework isn't a trivial one to answer.
But as someone who often has to jump into unknown codebases and get up-to-speed quickly
I'm not sure how long you've been in development but eventually you just learn to "expect magic" with frameworks and after awhile they all blur together. Show me the docs on the decorators, the pipeline, how it handles global context etc and it's all pretty much the same core concepts repeated across different languages.
To your later point, yes, frameworks should be a terse DSL. I expect a framework to make good "magic" assumptions that enable me to benefit from packaged functionality that would often be more verbose in lower-level implementations.
without needing to resort to external documentation to understand
That's not a thing. RTFM. Or a good code inspection tool if you really hate docs.
Your ability to pop open vscode and slap some JS in it to start a web-server in a few lines is the product of decades of SW frameworks that have made thousands of assumptions along the way to increase expressive power while tucking complexity under the hood. Feel free to provide a counter-example vs a generic statement.
Same as in this Rust example then. The Axum example is basically the same as its equivalent in JS. I've also seen similar abstraction in Python (Django, Flask), Ruby (Rails), Elixir (Phoenix) etc.
I will agree insofar as webdev seems to proliferate use of magic. ASP.NET core also uses a ton of it, which I hate as well. I'm not sure why that one particular area draws the people who love magic for their tooling, but it at least partly explains why I wish I could just hide in a backend hole and avoid 90% of it (Dependency Injection can be very full of magic as well but that's another tech tool I wish would go away).
> Rust developers generally like these properties, predictability and explicitness, instead of magic. That’s why I’m quite surprised that most Rust web frameworks went another route.
Ecosystems are rarely monoliths, and I assume that the people writing web frameworks in Rust are likely to have experience with (or at least be inspired by) more magical frameworks in other (usually more dynamic) languages, whereas the libraries that eschew magic are presumably written by people with prior experience in low-level languages. The fact that Rust has some amount of appeal to both of these crowds (not to mention the functional programming folks) makes it a bit of a melting pot.
The handlers are less hard to reason about then a lot of macros.
You don't have to understand the type magic which makes them work.
At least if you look at the more relevant frameworks (actix-web, axume) they basically boil down to calling a
fn foo(x: A, y: B) -> Result<Body, Error>
as roughly
let x = A::from_request(&req, &mut payload).await?;
let y = B::from_request(&req, &mut payload).await?;
foo(x, y)?;
and _nothing_ more.
There is no special handle first/last parameter different logic or anything.
They way they handle not copying the payload (request body) is by moving it (a reference to it) out of the request (or here payload).
The reason the performance for micro-benchmarking extractors is different depending on the order has less to do with the general design and more the typical "I reordered parameters and now optimizations trigger differently" problem. (Probably some in-lining and const propagation allowing eliminating some checks but just on one order, anyway not specific to web frameworks at all and can also happen with their approach, but probably less likely if I should guess).
No, indeed, the opposite. In Rust, these will surely be procedural macros, which comprise functions written in Rust which operate on Rust code, represented as token streams. The compiler compiles the macro, lexes your code, feeds it to the macro, gets a token stream back, parses that, and that's what it compiles.
Understanding a magic handler function involves understanding a particular subset of features of the Rust type system, and operating that mechanism in your head to see what happens.
Understanding a proc macro involves understanding arbitrary Rust.
As a relative newbie in Rust who recently dabbled with Axum, I wholeheartedly agree that the handlers and extractors are magical and hard for first-time users. Another issue I ran into that's not mentioned in the post, is the compile-time type errors are pretty cryptic, if you get it wrong. I was trying to use the new state extractor, and didn't have my state properly wrapped in an Arc and it was complaining to me in a way that was totally unhelpful. The docs mention there's an `axum-macros` crate which can help here, but I feel like if you need macros to get around confusing type errors, you're going against the grain pretty hard.
On the whole I still found experience pretty compelling, though, as I did eventually get a little web app running with great performance and very low memory usage.
The bad error messages are really a big deal. I like Axum, but this type-driven extractor technique leads to such awful messages for some trivial mistakes.
This was an interesting read for a Java / JVM language developer that has recently begun learning and using Rust in personal projects.I was curious how the magic was working begind the scenes. In the Java world, our most popular framework (Spring Boot) takes Magic to another level, since Java allows reflection, lots of magic is possible.
Luckily, there are saner alternatives on the JVM. If you read at "Your server as a function" paper from Twitter, you'll see that the composing of middleware and simple handler functions can be used to develop powerful stacks which are a doddle to write, reason about and test.
The SaaF design was used to design Finagle (Scala), and then it inspired http4k (Kotlin). There may be more that I'm not aware of. These libraries are often described as micro-frameworks - but this is misleading - it is only because the core is very small and can be easily extended by bolt-on modules.
Sadly, Filter is not a functional interface, so it's much clunkier. It's also not really clear why you would use a filter when you can do function composition; i suspect they added filters because previous Java web frameworks had them.
While extractors are sometimes doing copies, it is basically never in situations where it matters.
For example extractors on the json body will always _move the body out of the request_ (independent of the position the extractor is in).
Sure there is some copying for e.g. request parameters, but you very often do that anyway as you e.g. do some urldecoding, parsing and similar on the parameters, e.g. extracting a Uuid type from a `/user/9313ba53-955e-4ce1-8b53-de8f92656196/bar` path.
Similar if the extractor doesn't return the error type you want you can wrap it an map the error when extracting. Sure with actix-web and warp there are some error handling situations which they don't handle well, but that got improved on in axume.
Application state is shared across threads (or at least across time) so it's anyway in a `Arc` or `Rc` so you are just cloning the reference pointer but not the data behind it.
Lastly if you have a very special situation you can decide to just accept a `Request` moving (not copying) it into the handle and do the rest yourself by hand.
Without question the system has potential for improvements, but for now it has become the de-facto standard due to it being very convenient, does work well for many use cases and does not add a relevant performance overhead for a lot of use cases.
Now their solution is targeting backend WASM so they probably use will focus in single threaded WASM with some snapshot mechanics to easily sandbox each request so their requirements are anyway a bit different but looking into history routing macros where common in the past as far as I remember, and they are pretty much all gone by now.
But then they probably will go through a bunch of revisions of their system, so it's not like they can't add it back in later one.
EDIT: Just to be clear their solution is a fully valid reasonable approach.
Servlin contains no unsafe code. I picked dependencies to have as little unsafe code as possible. I wrote some unsafe-free deps: safina (async runtime), safe-regex, fixed-buffer, permit, temp-dir, and temp-file. I plan to replace other unsafe deps: url, async-net, and serde. Eventually Servlin's only unsafe will be in the standard library and `polling` crate.
Servlin supports single-file webapp deployments by including web assets in the binary. I plan to develop Servlin until it is as strong as nginx and can run reliably with no load balancer. And someday a good Rust unikernel will exist and we will run Servlin binaries directly on hardware. That will be the world's shortest, most secure, and most performant web stack.
Please contact me if you would like to help improve Servlin.
I agree with most of the criticisms of the "magic" functions, although I really have to disagree that the macro approach (with a totally custom DSL) for routing/authorization is less magical than the alternative. It also cripples IDE support. With a builder-pattern you can get smooth autocompletion and clear error messages, while macros usually sabotage error messaging and completely sabotage autocomplete.
Bigger picture: I think the reason so many Rust frameworks favor this kind of magic is to lower the bar (accessibility, but also development velocity). Being lower-level, Rust starts off with a disadvantage when it comes to quickly building out high-level application logic like you would on a typical web server, so when people are trying to expand its reach into areas like that, they try to close the gap by making their libraries as easy-to-use and boilerplate-free as they can. Rust's macro and type systems are the mechanisms it provides for eliminating boilerplate and generally making it act like a higher-level language, so that's what people use. And that makes well enough sense, even though it comes with trade-offs.
> ... I don’t know the “official” name for it, so I just stole the name from a recent reddit post (magical handler functions).
What's the difference between programming "magic" and "using the language to its fullest"?
Wikipedia has an article dedicated to the concept of magical programming. It starts with:
> In the context of computer programming, magic is an informal term for abstraction; it is used to describe code that handles complex tasks while hiding that complexity to present a simple interface. The term is somewhat tongue-in-cheek, and often carries bad connotations, implying that the true behavior of the code is not immediately apparent. For example, Perl's polymorphic typing and closure mechanisms are often called "magic". The term implies that the hidden complexity is at least in principle understandable, in contrast to black magic and deep magic (see Variants), which describe arcane techniques that are deliberately hidden or extremely difficult to understand. However, the term can also be applied endearingly, suggesting a "charm" about the code. The action of such abstractions is described as being done "automagically", a portmanteau of "automatically" and "magically".
I think that "true behavior of the code is not immediately apparent" is the key. The harder it is to figure out how something works, the more likely it is to be viewed as "magic."
Do the magical handlers in this article fit the bill? I'm not so sure. All that's going on is generics applied to functions. True, it's not as common as generics applied to other types. But it's also not fringe.
Maybe as a technique becomes more established, it starts to seem less magical. The author notes:
> Exploring this pattern was an interesting journey. I still can’t tell why it’s more popular than the alternative that contains less magic and gives more flexibility to developers. I can only assume that even Rust developers long for some magic and coolness in their tooling.
Maybe it's more popular because it's now widely used across Rust web frameworks and therefore expected as a convention?
> I don’t know the “official” name for it, so I just stole the name from a recent reddit post (magical handler functions).
I’d want a term mentioning that it’s type- or signature-guided or -driven. “Magical” is way too non-specific. How about “signature-driven handler functions”? Seems both accurate and fairly clear.
- Sandboxing of each request (is performant when the right kinds of snapshot+CoW technique is used)
- PaaS for running WASM compiled untrusted web applications, especially nice for "edge service style PaaS"
- Node style async of rust, can be a bit more simple then rust-async (but also likely more limited)
- Similar "compile once run everywhere" as JVM (but better due to limited interface and server focused?), i.e. you have the wasm runner compiled for each target but you need to compile the program only once and then you can run it on all kinds of platforms including x86_64, arm64 etc. Especially good for PaaS but also good for more flexible deployment pipelines in large companies.
And especially for many web use-cases, especially as run on a lot of "edge" services the drawbacks of single threaded WASM do not matter much. E.g. the workloads are of the kind which tend to work very well with WASM performance wise.
I think it's similar to the whole javascript/node concept.
In many ways, web development is like a single distributed application between client and server. So if you write your client code in Rust/Zig/C++/etc and emit wasm, wouldn't it be cool if you could share data structures and logic for that code on the backend that would handle client requests?
But you can share code between Rust compiled to wasm on the frontend and rust compiled to native on the backend. In fact I did this for a game where you can either play locally on the frontend or play networked with optimistic updates provided by the client-side but the rules enforced on the server-side.
> someone explain to me the point of backend webassembly for a web framework? you're running the compiler already, why not emit... actual assembly
Once you're in this industry, you stop trying to figure out why fads come and go. There really isn't any reason. It's all fashion. You'll go crazy asking "Why?".
Agreed. But maybe my reasoning is different to yours: it creeps me out when I can't tell what the type of something is. Macros fall under that category. Just give me something where values go in and values come out. Something goes in and arbitrary Rust code comes out? That's harder to reason about.
I disagree with the article a bit here:
> I also have the feeling that the “magic” word is just reserved for macros in the Rust community. If you manage to build the same features, but hide them behind the type system, Rust developers will mostly not consider it magic. In many cases I personally prefer well documented macros instead of crazy type system workarounds.
Crazy type system workarounds? Although I prefer the "traditional" approach, the "magic" approach just involves a function that takes a bunch of `FromRequest` arguments which then calls `T::from_request`, that's not all that crazy. The copying is unfortunate, but this isn't some kind of insane type hackery.
Maybe I've just been exposed to too much Haskell monad transformer soup to find this kind of mundane actually understandable stuff "crazy".
Types are the ultimate documentation, over-use of macros gives that up in favour of the macro developer hopefully writing comprehensive documentation, which doesn't always happen. If the compiler isn't making you write enforceably correct documentation (types), you probably won't.
But as someone who often has to jump into unknown codebases and get up-to-speed quickly, I've certainly been confused by framework magic before, which makes it almost impossible to reason 'up' from the concrete implementation of e.g. a request handler. Since the keywords to grep for aren't explicitly mentioned in the handler anywhere and the 'dispatch' side of things is hidden under mountains of abstraction, you usually have to resort to tutorials to understand what the hell is going on, and even then the codebase might use some extra-magical properties of the framework that aren't clear to anyone that isn't at home within it.
So it could be argued that this magic saves you code in exchange for a need for more documentation, at which point you might just be better off being more explicit in your code -- at least explicit enough so things remain greppable.