The most anticipated Django release for me personally. Async ORM access is a massive quality of life improvememt, same for async CBV. The later is also a blocker for Django Rest Framework going async.
Sure most CRUD applications work perfectly fine using WSGI, but for anyone using other protocols like WS or MQTT in their app this is kind of a big deal.
Seriously: why use Django for a REST-like service? I inherited an application that (probably) evolved from form-based to REST with a JS frontend, and I fail to see the point. There's a model, a serializer, handlers to implement relations, and multiple views per data type. It's so much overhead, and everything has to be kept in sync.
I went with Django Rest Framework when I used it for a project because it provided things I would otherwise spend says implementing out of the box. The one thing I definitely did implement for myself was filtering, the built-in stuff was kind of mediocre for our needs, outside of that I want to say we left pagination and everything else as-is. It was a bit of a nightmare to look up documentation on though I will admit, if you don't know the Google Buzzword Soup you got to look up. Previously to that I was using CherryPy and building my entire kitchen sink myself, so it had nice moments and bad moments, but it definitely saved me on time and effort.
You typically do Django + REST because you are already comfortable with Django or because you were already using it, although I could argue, you could consider making your API project stand alone and using CherryPy or even FastAPI for a much nicer experience. The downside there being you wouldn't be using Django's ORM. I really do wish Django's ORM would be spun off into a stand alone project that Django then imports, so anyone else can use it in other non-django specific projects.
I remember watching a beautiful talk from some pycon whose details I can’t remember about using django as if it was a micro-framework.
It basically boils down to rejecting the folder structure that django-admin sets up for you, importing stuff yourself and doing some basic initialisation/configuration.
Once you do that, django kinda somehow behaves like a micro-framework, with the significant difference that you can import and use the advanced features if you want/need to.
I guess I prefer to keep it light and use a true microframework like flask for this use case. I find I can similarly import Flask extensions without bringing in a full heavy framework
Proper Django Rest Framework should basically have zero view code. Treat Serializers as forms, point at them in the appropriate generic view (or viewset) subclass, and you are done.
If you are writing view code in DRF, you are probably doing it wrong.
I don’t think this is fair - the DRF views are a great starting point for a project but I don’t think I’ve ever worked on one where they’re totally sufficient without at least some modification in places if you don’t want to make use of signals (which are very difficult to debug!).
Even things as simple as adding logging to calls on certain end points will require you do provide your own view method (which might then call the mixin’s provided method).
I don't find it has a lot of overhead if you keep it simple. No need to use class based views for REST services, just add function-based views with the @api_view decorator.
You can then within minutes return JSON coming from your data model by also adding an additional decorator to make it render JSON output.
If returning data directly from managed ORM objects, you will need to serialize it, but you can easily build an API without doing that and just return a dict.
It all depends how it is written. While Django and REST is quite strongly opinionated, you can still make it difficult to maintain if you want to.
That being said, Django REST allows you to move fast. It's very very easy to knock out a REST API and data models plus some basic admin panel.
Once your API matures - you can then reimplement it in Go or other platform.
It really shines if you are doing simple CRUD APIs where you can lean on autogenerated routes, views, and serializers. Since you can get full CRUD for a model in something like 10 lines of code.
When you need to wire up custom serializers per model (eg specifying subsets of fields or making some read-only) and wire up non-CRUD actions then I think you can fall down a slippery slope where you write as much (or more) code than if you just used say marshmallow and flask to manually write your API.
Also you get the Django model admin for free, so for internal services you have a nice CRUD admin for operators.
I think it’s a great tool for accelerating your first 100-200kloc of API code. Beyond that it can start to creak at the seams depending on your usecase. (I’d make the same assessment about Django itself FWIW).
It's a really easy framework that just takes a few days to get proficient in. It is very well documented and because it comes with so much "overhead" you don't have to get your own team to make a set of different mistakes building the features as needed.
every python developer has accidentally made django at least once.
you start with "i will do in flask because it's easier". then you need to add authorization/authentication. and then templating, ORM, maybe some RESTful flask library to make endpoints a bit easier... and now you have django, but in a different way.
I just make normal money and I don't want to have lunch with Jeff Bezos. Maybe in this scenario you lose money each time you hear it and you're so in debt that Jeff Bezos has basically purchased you :P.
It makes sense as backend for a native iOS/Android app, and if you already have that REST endpoint then it can make sense to re-use it for the web-version of your app.
But if you're not doing an iOS/Android app (most websites?), then I generally agree it's a lot more work compared to form-based / server-side-templates (though sometimes can give a _slightly_ better user experience, _if_ done correctly.)
HTMX seems to be growing as a "REST endpoint" alternative, but again not as useful if you're also doing a native iOS/Android app.
In this day and age, mobile apps are _almost_ a must so REST is everywhere.
A well-implemented REST app is pretty nice, business logic divorced from all UI and API lends itself nicely as a comfortable and maintainable programming environment. More often than not, being RESTful from the get-go tends to have positive returns down the line.
> In this day and age, mobile apps are _almost_ a must so REST is everywhere.
This really isn't true. Mobile apps are far more expensive to develop (everything is doubled, you're dealing with periodic forced updates outside of your development schedule, etc.) and there are a ton of apps which don't need anything you can't do in a browser.
One of the earliest things to do on a project is agreeing on what level of complexity makes sense for you. If your project doesn't have a known hard requirement for mobile apps and correspondingly large team sizes, a classic Django app is likely to let you iterate an order of magnitude faster at lower cost overall.
I completely agree. I have been using GraphQL in Django with WebSockets and ASGI and it was a bit of pain in the ass to get everything working, but with this changes things will get much smoother.
`QuerySet` now provides an asynchronous interface for all data access operations. These are named as-per the existing synchronous operations but with an `a` prefix, for example `acreate()`, `aget()`, and so on.
> The new interface allows you to write asynchronous code without needing to wrap ORM operations in `sync_to_async()`:
async for author in Author.objects.filter(name__startswith="A"):
book = await author.books.afirst()
> Note that, at this stage, the underlying database operations remain synchronous, with contributions ongoing to push asynchronous support down into the SQL compiler, and integrate asynchronous database drivers. The new asynchronous queryset interface currently encapsulates the necessary sync_to_async() operations for you, and will allow your code to take advantage of developments in the ORM’s asynchronous support as it evolves. […] See Asynchronous queries for details and limitations.
## Asynchronous handlers for class-based views
> View subclasses may now define async HTTP method handlers:
import asyncio
from django.http import HttpResponse
from django.views import View
class AsyncView(View):
async def get(self, request, *args, **kwargs):
# Perform view logic using await.
await asyncio.sleep(1)
return HttpResponse("Hello async world!")
I don't fully understand the sync_to_async() operation here. So the underlying database is not async, then is Django just tossing the sync DB call onto a thread or something?
Yes, it is just being put on a thread. A future Django release will use async database drivers. Django is very incrementally adding support for async. I don't believe they have started work on async templates yet, which will be one of the last major areas before the framework is fully async.
Yes, they're working from the top of the stack down, so first it was views+middleware, now it's the top layer of the database stack, and in future releases they'll work on pushing native async lower and lower in the stack.
Basically every function call that does db-queries needs an "a"-prefixed version. aget(), acreate(), aget_or_create(), acount(), aexists(), aiterator(), __aiter__, etc. And in future versions they'll work on the lower level, undocumented api, like _afetch_all(), aprefetch_related_objects(), compiler.aexecute_sql(), etc.
Eventually you'll probably need to switch database drivers to something that actually supports async.
Why do they need Async templates? When a template renders it should be pure data and fully on the CPU. It seems Async templates would encourage a bad behavior of doing n+1 queries inside the HTML.
My immediate thought would be custom template tags. Like `{% popular_articles %}` would have to connect to the DB and load the articles. I am sure there are other reasons too.
Also, async views allow for better scaling. Say your view needs to make an API call to a third party service within a request-response cycle. Currently, this will block an entire worker/thread while doing a blocking request.
With async, we can use async HTTP libraries and scale these WAY better.
Or just run 20 workers and let the OS handle async. OSes are pretty good at this. Async is pretty hard to reason about.
Async is beneficial when you make multiple blocking (http) request to a resource at the same time and later fold that into one response. Parallel stuff. Or when you are forced to run a single thread. Or when you are memory bound.
Reality is that most requests depend on previous requests quite often. In the latter case there is no benefit in terms of speed for the user.
Might be unpopular but still good to stick to it. I tend to associate excessive enthusiasm for pythons async await framework as a sign of immaturity and potential for doing things without understanding them fully. I’d never trust async code from anyone except the best python engineers. Most of them out there can’t even debug linear code, adding async to the mix only makes it worse. At least you can trust the time stamps.
>I tend to associate excessive enthusiasm for pythons async await framework as a sign of immaturity and potential for doing things without understanding them fully.
Couldn't it be the inverse? I.e. your inexeperience with async code in other languages, and not understanding it fully, translating into "new thing aversion"?
I think my experience in async in other languages has shown how rough async in Python is.
Unless async is the default python, it will always be an after thought and introduce needless complexity when working on large projects that require a lot of external libs.
Or when accidentally some blocking database driver is used, even when django's ORM is async, a query with mongodb's driver is not async. Or just the filesystem... This harms performance badly, especially if the idea was to run only a few threads.
And indeed, there's an important caveat in the Django 4.1 docs;
Note that, at this stage, the underlying database operations remain synchronous, with contributions ongoing to push asynchronous support down into the SQL compiler, and integrate asynchronous database drivers....
> Or just run 20 workers and let the OS handle async.
That just means you can only handle 20 concurrent requests before performance drops off a cliff. If the third-party service you are talking to is slow, then you’ll just end up with 20 workers all waiting for the service to respond, while other requests pile up. With async, those workers could still be handling other requests. Adding more workers because you are blocking on i/o works for low traffic services, not for anything remotely busy.
> Adding more workers because you are blocking on i/o works for low traffic services, not for anything remotely busy.
Yes.
That just means in very bad cases, you can only handle 1 "concurrent request" before performance drops off a cliff. If the third-party service you are talking to is that slow, then you’ll just end up with bufferbloat (high latency in hidden queues) waiting for the service to respond, requests piling up.
With async, unless you know what you're doing and handle back pressure, work pile up even worse, the service stops working and you get 0 throughput instead.
Agree! Up the number of workers to 40, fix your application in another way... or indeed go with async stuff! ;-)
I was just saying that async quite often is not of much help and just complicates things. Of course it has its use! Your case is a great example of converting a django view into an async view.
Why would you block a process that could use CPU cycles to do something useful just to wait on I/O? With the right primitives async is not "hard" nor inconvenient. Maybe you're referring to concurrency?
No I am referring to async, really. Accidentally blocking operations like open() or psycopg or mongodb or some included library which (accidentally) uses any of that stuff. Basically any library requiring requests is not usable. Or worse: open().
It can be hard to determine if something is doing something blocking and if it is, it is hard to debug. Especially in python world which, unlike javascript, was not async from the start.
I don't understand, are you saying async is hard or that a poor implementation is a source of issues (e.g. async function that calls sync primitives)? Because I totally agree with the latter but I have no idea what "hard" means. Very little changes in terms of syntax (assuming support for async/await ops) or reasoning.
> Or just run 20 workers and let the OS handle async. OSes are pretty good at this. Async is pretty hard to reason about.
def slow_sync_view(request):
# these requests won't be executed in parallel;
# async version could eliminate this extra latency
foo = requests.get('https://www.google.com/humans.txt').text
bar = requests.get('https://checkip.amazonaws.com').text
return HttpResponse(f'{foo}\n{bar}')
I think you are being down voted because a straight port of this code to async would still run one request after the other.
You would have to queue one task for each request in the event loop and then await for them both to gain some parallelism in the I/O section of the code.
Yes, if you do WebSockets, async really is a must if you don't want to have one thread per active connection. Each thread takes 8mb of virtual memory, not sure how much actual memory.
It's also nice for a backend that's doing a lot of proxying apis to another backend.
Sure most CRUD applications work perfectly fine using WSGI, but for anyone using other protocols like WS or MQTT in their app this is kind of a big deal.