I've been using httpx 0.9.3 in production now for a couple months. I switched from requests when I realized I needed async support and it has been a dream to use.
The only issue I've run into has been with my attempt to reuse the same AsyncClient to make multiple concurrent requests to the same remote host. It looks like this issue may have been fixed in 0.10 or 0.11 so I'll be upgrading soon to check.
Also, be sure to check out the other fantastic projects by Encode. https://github.com/encode I stumbled upon httpx after using Starlette and Uvicorn for one of our microservices and and been pleasantly surprised by how easy to set up and use they are.
I'm currently using aiohttp. There is features I don't find in butterfly docs like : limit connection, limit connection per host or per time frame.
Do you know if those exist in butterfly?
I'm already deep in aiohttp and it's not an easy task to learn an async client (at least in my case, but I'm no dev) so if I do switch it would be for more features (retry option on exceptions, limit requests per host and per time frame...). But that's only my opinion.
I prototyped moving to aiohttp from requests/multiprocessing. The speedup was amazing. Saw something like a 60% reduction in runtime for our use case. Only reason the code hasn't gone live: we currently use requests_negotiate_sspi for authentication, which sadly isn't supported by aiohttp. Not sure if httpx supports it. Looks like it might. Planning on giving it a shot next week.
If you’re interested in trying httpx you’d be very welcome to raise an issue an issue related to NTLM/Negotiate authentication - it’d be really helpful for us to work through that and figure out if our auth API is sufficient for implementing that, or if there’s anything we’re missing.
This is a huge win compare to requests. AFAIKT requests is too flexible (read: easier to misuse) and difficult to add type annotation now. The author of requests gave a horrible type annotation example here [0].
IMO at this time when you evaluate a new Python library before adopting, "having type annotation" should be as important as "having decent unit test coverage".
The huge win against requests is that Https is fully async.. you can download 20 files in parallel without not too much effort.
Throughout in python asyncio is amazing, something similar to node or perhaps better... That's the main point.
And the tests reference a timeout argument. If that doesn't work, I wonder how much work it would be to send a PR (instead of just talking s to Ken and not contributing any code)
> "having type annotation" should be as important as "having decent unit test coverage".
I mean this is pretty spot on IMO. I've worked with many languages, and have concluded that having a powerful type system catches soooo many bugs before you even try to run the code.
And they're usually "stupid" bugs too, forgetting to sanitize inputs etc. Even worse is when a language tries to be "smart", so you end up with "1" + 2 = "12" and no errors at all.
Type systems and static checks in general are great. What do you think of the example type in the post above? It seems like the type system might not be expressive enough to handle existing APIs, that’s something which was hard for TypeScript too, but has slowly been getting better. Perhaps that will be the case in Python too.
Personally, I use type aliases in cases where I need to describe very dynamic types that make intuitive sense but are too wordy to pattern-match visually. So you might decompose that example like:
You theoretically lose some "glance value" because now you have to look in two places for the type...but in practice, I think you can figure it out a lot easier than the original example. Obviously you don't want to do this in simple cases, but it can make pathological cases like the above a lot easier to chew on.
Some parts of that example seem pretty silly. URLs for example: he describes how just about anything can be passed in as a URL, and requests attempts to call __str__ or __unicode__ on it and then parse it. Considering (essentially?) all types in Python have __str__, this is a perfectly reasonable place to use the Any type. The issue isn't with the expressiveness of the typing system, but with the absurd flexibility of input handling by requests.
‘object’ is the base class of all types: you can put anything into an ‘object’, and you can only do very generic operations like str() on what you get out of an ‘object’ without further checks like isinstance().
‘Any’ is an unsound escape hatch that disables type checking: you can put anything into an ‘Any’, and you can do anything with what you get out of an ‘Any’, and the type checker will make no effort to stop you from doing something wrong.
Actually the proper way to handle is to use protocols. They provide already SupportsBytes type, and you can similarly create custom SupportsStr, but IMO it is silly to do that. This approach increases chance of bugs, and it's just easy to use str type and for developer to wrap such value in str(). I believe that's why SupportsStr was not introduced.
You should check out the Literal type in Python. It provides for some of the stupider things in libraries, like return types that vary based on the value of an input parameter, as in Python’s open() method, which might return a text stream or a bytestream based on the value of the second parameter.
> And they're usually "stupid" bugs too, forgetting to sanitize inputs etc. Even worse is when a language tries to be "smart", so you end up with "1" + 2 = "12" and no errors at all.
The last one is no longer possible in Python 3 and will throw an error. That's another thing I am glad they fixed.
That's from 2015, his second complaint about interfaces vs inheritance is solved in recent versions with Protocols. Compatibility with 2.7 shouldn't be argument today either.
The complex union is as you say more a sign of a too flexible api rather than problem with type hints, the type hints just brings the root issue to the surface. I mean, why would you accept either a mapping or a list of tuples as input? Just let the user call dict(..) on the tuple-list first if they have it in such format? The documentation doesn't even mention that lists are ok for headers, only dicts: https://2.python-requests.org/en/master/api/#main-interface.
The file-tuple api with various length tuples is perhaps valid and the most convenient way to implement such options, but it's still an exceptionally unique api which requires exceptional type hints, it can be made slightly simpler which chc demonstrated above.
This framework looks incredibly simple and powerful. I’ve coded a bunch of these features manually in the past. I think FastAPI is going to earn a spot in my next project!
Autocompletion everywhere, and from that, data validation, serialization, documentation. Dependency injection, OAuth2 stuff. Web standards. https://fastapi.tiangolo.com/features/
In some high-level languages even BSD sockets (and many other POSIX functions) aren't in the standard library, and there are various wrappers to provide "ease of use" and integration with a language's runtime system; plenty of complexity (and alternatives) even at that point.
RFC 2616 (HTTP/1.1, 1999) may seem manageable, but it's much more than just posting data and getting a response, and IME many programmers working with HTTP aren't familiar with all of its functionality. Then add TLS with SNI, cookies, CORS, WebSocket protocol, various authentication methods, try to wrap it into a nice API in a given language and not introduce too many bugs, and it's rather far from trivial. But that's just HTTP/1.1 with common[ly expected] extensions.
Edit: Though I think it'd also be controversial to add support for particular higher-level protocols into standard libraries of general-purpose languages, even if it was easy to implement and to come up with a nice API.
Plenty of reason why it's hard to ship in standard library. Here's some off top of my mind:
- Should the library includes its own CA store, or use the system's CA store? These kind of library often include their own CA store (since they changes often), and httpx seem to use 3rd party lib to handle that (certifi). This is hard to do in a standard library for variety of reasons (users rarely update their python installation, system CA store is not always available/up to date, etc).
- While the http protocol itself is pretty stable, some part of it are still changing overtime. Things like compression types (brotli is gaining traction these days, and we might get a new compression types in the future), new http headers added, etc. Security issue also show up all the time. The user will want tighter release schedule than python's so they can get these stuff sooner. The situation is even worse for users that stuck in a particular version of python for some reason since they now won't have access to these new update ever.
> Should the library includes its own CA store, or use the system's CA store?
The CA store should be a configurable option, and one of the supported options should be the system CA store.
> The user will want tighter release schedule than python's so they can get these stuff sooner.
Ruby is moving stdlib to default and bundled gems, which addresses this. There's no reason that “delivered with the interpreter” needs to mean “frozen with the interpreter”.
> The CA store should be a configurable option, and one of the supported options should be the system CA store.
It's more complicated than that, especially if you aren't on Linux.
On both the other two big general purpose platforms (Mac OS, Windows) the vendor provides a library which implements their preferred trust rules as well as using their trust store.
On Linux what you usually get is the list of Mozilla trusted root CAs and you're left to your own devices. Mozilla's trusted list is IMNSHO a shorter more trustworthy list than supplied by Apple or Microsoft, but it misses nuance.
When Mozilla makes a nuanced trust decision for the Firefox browser that decision doesn't magically reflect in an OpenSSL setup on a Linux server. Say they decide that Safety Corp. CA #4 can be relied upon to put the right dates on things, but its DNS name checks are inadequate and no longer to be trusted after June 2019. Firefox can implement that rule, and distrust sites with an August 2019 cert from Safety Corp. CA #4, while still trusting say, the Safety Corp. CA #4 certificate on Italian eBay from March 2018. But there's no way for your Python code to achieve the same outcome relying on OpenSSL.
Python's key people seem to think that it's better for Python to try to mimic what a "native" web browser would do because that's least surprising. So on Windows a future Python will trust Microsoft's decisions, on macOS they'd be Apple's decisions and only on Linux will it be Mozilla decisions. Today it's Mozilla's trust store everywhere.
Hypothetically in the ideal case you'd have your own PKI and you'd have all the necessary diligence in place, hire your own auditors, maybe even have contractors red-teaming the CA you trust for you - but we don't live in a world anything like that, most people are implicitly reliant on the Web PKI and probably tools like these needs to accept that.
HTTP/2 was only standardized 4 years ago. HTTP/3 is being actively developed. That's actually not really stable on a language's standard library timespan, IMHO.
It's simple. Python was created before HTTP even existed. Since then a lot of things had changed, and once you create an API, it is hard to rewrite it when new use patterns emerge.
It's easier for a 3rd party package to come up with a better api, because they can start brand new. Also when there's a radical change it is easier for a new 3rd package to take over. Httpx is example of such evolution, although not due to changes in http but this time changes in Python, it makes use of new functionality that's harder to implement in requests, mainly adding async support and type annotations.
Of course it could be added later (there's urllib that almost no one uses), I meant that Python is older than HTTP and HTTP evolved a lot during that time.
Once you create an API, it's hard to change it. HTTP initially was very simple and evolved over time. Things like REST, JSON encoded messages, cookies, authentication (albeit rarely used), keep-alive were added incrementally. Today's HTTP is used completely different than 30 years ago. Python already had urllib, then urllib2 which was renamed back to urllib in Python 3, but its API is still behind how HTTP is used right now.
httpx makes a pretty good first impression on me. The homepage provides examples, key selling points, install instructions, and links to any further reading one could hope for. However, I am missing one thing: The features this offers over the popular requests library do not seem to require httpx to be a competitor, but an extension or fork. There surely must be some major incompatibilities that allowed this library to do something fundamentally different, right?
> The features this offers over the popular requests library do not seem to require httpx to be a competitor
Yes it has to be. Requests is not as great as everyone thinks it is. It's API is simple, sure, but when you need more advanced features (doesn't even support HTTP/2 AFAIK) like timeouts, proper exception handling (which you cannot do in requests)) it actually sucks.
I know this is unrelated, but probably a lot of Pythonistas here and I've been wondering: what is the async web framework of choice for you guys today? As for DBs, still SQLAlchemy? What about a prettier (js) alternative?
I've been using FastAPI https://github.com/tiangolo/fastapi which is built on top of Starlette as my main async framework and Uvicorn as my primary web server.
Starlette and Uvicorn are both made by Encode https://github.com/encode and in my experience they consistently put out quality stuff.
I used aiohttp and it is quite good and solid (I also like that it comes with type annotations, which enables autocomplete in PyCharm), I did not have chance to compare it to FastAPI.
Regarding database access, my recommendation might not be popular, but I really like asyncpg[1]. They basically dropped DBAPI and instead created their own API that functionally matches PostgreSQL, giving you full control and increasing performance.
As a result of that you probably won't be able to use ORMs (there's SQLAlchemy driver, but as I understand it is hacky and you are losing the performance benefits).
Personally I compared my code with and without ORM and there's no change in number of lines. If you use PyCharm and configure it to connect to your database (I think that might be a pro feature) it will then automatically detect SQL in your strings provide autocomplete (including table and column names) and provide highlighting, removing all the advantages that ORM provided.
I've moved to Trio over asyncio (I did plain old async for a couple years and Trio makes a ton of sense)
Quart-trio over Flask (just to get a Trio-friendly flask-a-like server) - plain old aiohttp worked really well too. It takes a bit more roll-your-own work, but you get exactly what you want.
peewee over SQLAlchemy (less committed to this change, but peewee has been fine so far and is much more streamlined) I'm mostly just using SQLite. the async version of the ORM looks pretty new, i'm not using it yet.
Looks like this does not include certifi [0] and loads system certificates by default. This is a breath of fresh air to see, because so many packages want to use their own certs and have a custom system to override it to use system certs.
Edit: Well, looks like it does use certifi. But my grumble still stands, I don't understand why does everyone want to mess with your certs.
I want to mess with certs in Python so that my web crawler can actually access the whole web. If you don't talk to a wide variety of hosts, you probably haven't noticed that it's broken for 1%.
If you work in a corporate environment, you would probably notice that systems that insist on bundling their own certs without an easy to activate option of using system cert store are broken. (And even if the library has an easy to use option, if it's easy to not expose it, much software built on the library will still be broken.)
People should be empowered to substitute cert stores, but the system store should be the default.
One thing requests still has going for it is that it’s used under the covers by a lot of client libraries with a dynamic import, allowing you to pass in a session object to control things like connection pooling and custom headers.
I do look forward to httpx becoming the new “standard” though. Tom is a great developer, and his ecosystem of tools are going to have a really big impact in python web dev over the coming years.
Too bad they carried over the biggest nuisance from requests, having to call raise_for_status() after every single request. One extra line everywhere and another thing that can be forgotten and cause strange errors when you least expect it.
I like async support, but always, in async examples, doesn't the await keyword just block anyways?
I like to see the usage of async inside of a main loop, where other things can be processed while waiting for the async response to come in.
Further, not all applications have the idea of an event loop, so async may not be needed, but it's useful to have the option.
I use async operations to multiplex operations in a queue. The linux scheduler can handle the execution time slices for me, I'm not going to build a scheduler, but the queue controller's role is to accept jobs and handle timeouts and results of each async operation.
The await keyword only blocks the execution of the coroutine in which it is used. It releases the event loop so that other coroutines can continue processing while the result of an awaitable is being fetched.
Comparing the code samples, it looks like it's easier to spin up testing with Httpx than Flask, but that's a superficial conclusion. I don't have experience with Httpx so take this with a grain of salt. Based on the comparison, I'm going to play with Httpx for testing the next time I use Flask because the simplicity looks rad.
Based on similar experience with other tools, that possibly means that Httpx is great for simple testing but if you need to go deep it's better to use the framework provided. That's an assumption, though, so I'd love to hear more from others.
Flask’s sample code is doing a bunch of things like spinning up db and stuff and setting up a pytest fixture, so it’s not a fair comparison. You can replace
with httpx.Client(app=app) as client:
...
in the httpx sample code with
with app.test_client() as client:
...
for flask’s builtin test client and the rest is basically the same. Now that’s a fair comparison and neither is simpler than the other.
I guess one advantage of httpx is that developers might generally be more familiar with the requests response object API than the werkzeug response object API.
My project has a dozen or so libraries, many of which require different and conflicting requests, urllib3, etc., libraries. I have to setup the packages very carefully to ensure the right module gets the right version of its HTTP library and it has generally turned me off to Python HTTP libraries altogether. For my own code, I stick to the basic built-in libraries, regardless of how difficult to use they are.
I use intensively requests-cache, that add local caching (in sqlite) for python requests and is configurable (should it cache non 200 response, should it cache POST ...) and it's amazing when scrapping. I didn"t find a similar companion package for httpx.
I immediately wondered the same, and I think not. The upper wings are not at a 90 degree angle to the body, so alive and well. Apart from being a black and white drawing of course.
The only issue I've run into has been with my attempt to reuse the same AsyncClient to make multiple concurrent requests to the same remote host. It looks like this issue may have been fixed in 0.10 or 0.11 so I'll be upgrading soon to check.
Also, be sure to check out the other fantastic projects by Encode. https://github.com/encode I stumbled upon httpx after using Starlette and Uvicorn for one of our microservices and and been pleasantly surprised by how easy to set up and use they are.