You don't need to be Facebook or Google to have more than one service in your infrastructure that needs to authenticate a user's existing session without forcing the user to log in again. Sharing the session across multiple services is its own distributed systems problem with numerous security implications to be aware of and bearer tokens might be a good alternative.

If all you have is a single monolith web app that is the identity provider, makes all authentication decisions etc then yes, you don't need JWTs probably. There is a huge gap between that and being Google/Facebook.

Apart from that, Google and Facebook don't even use JWTs between the browser and backends after the initial login but actually do have some sort of distributed session concept last time I checked.

Thank you. This middle ground between hyperscaler infrastructure and super simple web apps is where most of my career has been spent, yet the recent trend is to pretend like there are only two possible extremes: You’re either Facebook or you’re not doing anything complicated.

It has an unfortunate second order effect of convincing people that as soon as they encounter something more complicated than a simple web app, they need to adopt everything the hyperscalers do to solve it.

I wish we could spend more time discussing the middle ground rather than pretending it’s some sort of war between super simple or ultra complex.

If we're talking about a web session, time-limited randomly-generated session tokens that are stored in a DB still work fine. If you really need it, put a caching layer (memcached or redis or valkey or whatever) in front of it. Yes, then you've created cache invalidation problems for yourself, but it's still less annoying than JWT.

If we're talking about authenticating API requests, long-lived randomly-generated auth tokens stored in a database work fine, generally. (But allow your users to create more than one, and make rotation and revocation easy. Depending on your application, allowing your users to scope the tokens can also be a good thing to do.) Again, put a caching layer in front of your database once you get to the scale where you need it. You probably won't need it for a while if you're sending your reads to read-only replicas.

(Source: worked at Twilio for 10 years; we definitely eventually ran into scaling problems around our user/auth DB, and our initial one-auth-token-is-all-you-need setup was terrible for users, but these problems were fixed over time. Twilio does use JWTs for some things, but IMO that was unnecessary, and they created more headaches than they solved.)

I'm not saying no one ever needs JWTs, but I think they're needed in far fewer circumstances than most people think, even people who agree that JWTs should be looked upon with some skepticism. If you need to be able to log people out or invalidate sessions or disable accounts, then JWTs are going to create problems that are annoying to solve.

(One possibly-interesting solution for JWT-using systems that I haven't tried anywhere is to do the reverse: don't cache your user/auth database, but have a distributed cache of JWTs that have been revoked. The nice thing about JWTs is that they expire, so you can sweep your cache and drop tokens that have expired every night or whenever. Not sure how well this would work in practice, but maybe it's effective. One big problem is that now your caching layer needs to be fail-closed, whereas in a system where you're caching your user/auth DB, a caching layer failure can fall back to the user/auth DB... though that may melt it, of course. I also feel like it's easier to write logic bugs around "if this record is not found, allow" rather than "if this record is not found, deny".)

You just (somewhat handwavingly) described what Google and Facebook are doing. You might not need to build this globally highly available distributed session store, JWTs might be an ok solution for your use case too (because you are not Google or Facebook) - or not. It depends on what your requirements are. AuthN across services is somewhat complex in any case, I don't think there is an easy way around it without making tradeoffs somewhere. JWTs are a great tool to consider here.

This works fine for a single service but you’re replying to a thread about the middle ground of multiple services. It’s an anti pattern to have every service talk to the same database just to authenticate every request.

By the time you add a caching layer you’re truly better off using an off the shelf oidc id provider and validating the id token claims.

Postgres has insanely good read performance. Most companies and services are never going to reach the scale where any of this matters, and developer time is usually the more precious resource.

My advice is always, don’t get your dev team bogged down supporting all this complicated JWT stuff (token revocation, blacklisting, refresh, etc) when you are not Facebook scale / don’t have concrete data showing your service really truly needs it.

A simple JWT implementation isn't that complicated, but you have to accept some limitations.

For mostly-read flow like authentication, a centralized database can scale really well. You don't even need postgres for that.

If you have mutable state, JWT can't help you anyway.

JWT start make sense only when you are doing other hyperscaler stuffs and you can reuse part of those architecture

I was inspired by https://philip.greenspun.com/panda/ circa 2001 to develop a complete user management framework based on that kind of cookie which had the advantage over other systems that the “authentication module” it took to get authentication working in a new language was maybe 40-100 lines of code. Software like PHPNuke that combined second or third rate implementations of apps all in the same codebase was the dominant paradigm then, the idea that you could pick “best of breed” applications no matter what language you were using was radical and different.

I used the framework for 10+ projects, some of which got 350,000+ active users. As an open source project it was a complete wash. Nobody got interested in user management frameworks (as opposed to writing your own buggy, insecure and hard-to-use auth system in a hurry) until around 2011 or so when frameworks based on external services all of a sudden popped up like mushrooms. Seemed like the feature I was missing was “needs to depend on an external service that will get shut down with the vendor gets acquired”

Bullshit.

If you’re having trouble with that, you’ve got bigger issues. Any regular work queries will take longer, and so it’s not even a meaningful area of concern if you broke down a request from end to end on a flame graph.

Yeah, so? They don’t have to be talking to the same system, and in fact it was literally what you called bullshit to originally.

> If you’re having trouble with that, you’ve got bigger issues.

That does absolutely nothing to changing the fact that a SPoF is still an anti-pattern that should be avoided.

For that matter…

> A in-memory cache on top of session retrieval is so trivial and adds so few microseconds that it’s imperceptible even at large volumes of traffic.

Also does absolutely nothing to change that fact. You have done nothing to actually elaborate on why it’s totally not a horrendous idea to have everything communicate to the same database. Just because there’s a caching layer does not mean that fresh data wouldn’t be available if a SPoF goes down, which, once again, is the whole point here.

This statement that doesn't mean anything.

If you want people to believe what you say, you have to actually explain why you think something. Just saying it does not somehow make it true.

Be better.

But that also means that you can have JWTs that are used as "random token" for most of your app, cost to produce them isn't high, and only make use of the additional capacities for instance when

- when you want to check signatures (e.g. reject before hitting your application layer)

- store non sensitive base64 data that you want before restoring the session

Creating and handling JWT is only as costly and complicated as you want it to be, so there's IMHO enough flexibility to have light use with very few penalities for it.

It’s not necessarily easier than just passing the jwt, but with our internal setups where when you first pass through the authorisation system, our traffic on your behalf is secure it doesn’t really warrant a reason to decode your token multiple times rather than simply passing your access permission claims.

We do still pass your jwt between isolated “products” where your access request doesn’t pass through dapr, but rather back through our central access gateway and then into the other “product”. A product is basically a collection of related services which are restricted to be a business component. Like a range of services which handles our solar plants, and another business component which handles our investment portfolios, and so on.

It's shocking how often this advice isn't followed. We often see it with non-tech companies who nonetheless deliver services over the internet.

Every cache has TTL, so you just set the TTL of the entry to the expiration date of the token you are caching. No need for nightly cleanups.

Revocation of tokens can be done for a simple logout operation, in which case the stakes are low, but more often it is the "pull the fire alarm and get that user out", and in that case it should be reliable.

My understanding was that this is the ENTIRE benefit of JWTs (over plain session token): they allow you go from a allowlist to a blocklist, which is more efficient at really large scales because you only have to store expired sessions (until their time-limit expires) rather than every session).

And if you're not doing this then there's no point in using JWTs (which will be the case for most people).

Are there any other benefits I'm missing?

I don’t know of any companies that even do this. As far as I know, most use cases store nothing, except for of course the client storing the response.

Thing is, token refresh service can be stateless, but adding a revoke service basically kills JWTs main advantage, since every time we check its validity, we need a query to see if its been revoked.

But since such a revocation list is going to be short (usually 0 entries, dozens at worst), it's trivial to replicate across all the auth service nodes (which can as well be worker nodes) or keep it in Redis replicated per DC, with sub-millisecond lookup times.

Things get harder if you want a feature like logging out other sessions, or just an explicit logout on a shared computer (think about business settings: stores, pharmacies, post offices), you may have to have larger revocation lists. This may still not be a problem: a million tokens is a few dozen megabytes, again, a per-DC replicated Redis cluster would handle it trivially and very cheaply.

I'm of a more gradual opinion regarding this - say you operate a movie streaming service and control access to movies via JWT. It's not a problem if an attacker has access for two more minutes than intended.

If you are talking to a single client, I think checking the remote IP address and encoding it in the token might work to see if the token is not stolen, but don't quote me on that.

I get that this has conceptual appeal, but I doubt this makes any difference in real life. Unless you have some very sophisticated infrastructure, it takes many minutes to discover the issue and then many more minutes even to decide what to do about it. A few extra minutes to cut off access is probably not going to make a big difference one way or another.

When I had a chance to design a token-based authn/authz system, we had two types of tokens, general access (with hours of expiration, mostly read-only access) and privileged access, with expiration time set to a minute or so. All auto-refreshed on use, all separately revokable.

Seems to me that for most applications, the irrevocable 5-minute token seems "good enough".

To revoke a token:

1. issue a new token to the revoker that is issued at current time (if business rules require revoker to be logged in).

2. set user inb to current time - 1 second with a TTL of longest issue * 1.5.

3. Add user to bloom filter.

4. upload bloom filter to s3, every service downloads this every 5 minutes.

5. Then on request, check bloom filter. If the user id is in the bloom filter, check with revocation service that inb > issued time.

This is probably less than five hundred lines of code and pretty easy to maintain.

It also makes e2e testing very easy since you should be logged out after pushing that button.

But yeah, if you have a revoke service, might as well just use session keys.

Edit: typo

Back in the day we used memcached for our primary store for all sorts of ephemeral things. Including user sessions.

Actual current example: Small company acquired another with a complementary product, but on an entirely different tech-stack and cloud-hosting.

Product owners want customers logged into Service X to be able to follow a link to Service Y and not log in again, and vice-versa.

How is this better than JWT if we have 30 microservices called from front-end?

100% this. I am tired of you don't need microservices, you don't need JWT, you don't need Kubernetes, you don't need ElasticSearch, you don't need IAM, you don't need Redis, you don't need Mongo and everything should stay in one SQL database.

Things are being used not because they exist, because people want to be fancy or because they don't have something better to do. Things are being used because they solve problems and do so with least effort possible.

There's also "blinker-driven development" where people pick the solution based on their own personal set of hammers rather than, again, something more suitable.

(There's loads of these though - e.g. "optimisation-driven development" where the solution MUST GO BRRRRR even if the problem could be fixed by Frank typing in "Yes" once a week. "GOF-driven development" where everything has to rigidly fit into a GOF pattern regardless of whether it actually does. "Go-driven development" where everything has to be a interface and you end up reading a method called Validate which calls an interface method Validate which calls an interface method Validate which calls an interface method Validate and you wake up screaming every morning because why just wtf why please help me pleasehelp)

I don't say what you describe doesn't happen, but it's my impression that most people try to adopt solutions that minimize costs and development time (which also translates to money). 99% of the time it's not "do the best thing to satisfy solve this problem" but it's "solve this problem as fast as possible, without adding additional costs and using as few developers as possible".

Agree with that but from my experience that's more like 20% of the time. The rest is the various kinds of bullshit development where people are padding resumes, having boss's pet hobby horse forced on them, latest shiny flimflam, etc.

(A decent chunk of that 30 years has been contracting and that tends to be at places with problems which might be biasing my sample set.)

Or people in this industry are geeks and curious and like to try new stuff and technologies just for fun?

That's the case for the majority of people I've worked with.

Sure but do that at home or on POCs. Not on production code.

The whole codebase is riddled with the same kind of layering but we do now have guidance about doing stuff like that ("DON'T") and a plan to simplify some of the worst offenders (like the multi-layer `Validate` hell hole.)

At least on different schemas.

That and don't let one concern access data from another, or you'll have to coordinate schema changes between those different concerns.

God, how hard is SELECT * WHERE …, seriously.

You need to share a session across websites? Wow! Connect to the database holding the sessions.

Boring.

Standard solution is query a session table on from a single location, and once you actually start to need to trim request time, it’s not even the first place you look.

* everything is located physically close together (good luck reading from a DB table in Singapore from a service in Europe for every request)

* you have few enough client services that want to do this that the number of db connections does not become a problem (this is 100 by default for Postgres, you might need to tune this or deploy proxies, etc)

* high availability is already taken care of (you don't want that one DB server to bring down everything in case of a failure)

Should a project use sessions or JWTs? One isn't right or wrong, it all depends on the context of the project.

Do you maintain a database of JWT session tokens for refresh and revoke?

Do you have a real session that you load for every user every request anyway?

If the answer is 'yes', then the answer to 'use JWT' isn't 'it depends'.

It's no.

JWTs have absolutely been over-hyped for the last 8-10 years, but they do have a use and you don't have to be at the scale of Google for it to be the right approach.

Software isn't as simple as creating a checklist of a few basic categories and saying there is always a right or wrong answer. The answer should be "it depends" because there are many more factors at play when deciding something as fundamental as authentication and authorization.

For starters, the blog post leads with the premise that the answer is always "No". Even ignoring that as playful banter, the checklist is way to broad to be universally true.

> You wanted to implement log-out, so now you’re keeping an allowlist of valid JWTs, or a denylist of revoked JWTs. To check this you hit the database on each request.

> You need to be able to block users entirely, so you check a “user active” flag in the database. You hit the database on each request.

> You need additional relationships between the user object and other objects in the database. You hit the database on each request.

> Your service does anything at all with data in the database. You hit the database on each request.

These all assume that I authorization data and that the user data lives in the same database. I may store my own user object with a unique ID that references an authenticated user, while the author flow is actually managed by an entirely different service (whether first party or third party). I also may have my user data stored in a separate database, say for legacy reasons or data security/integrity requirements. Not every database is crested equal, its totally possible that one database is a better fit for user data while a totally different database is needed for application data.

Coming up with a list of broad scenarios and claiming that if you check any one there is never a reason to pick JWTs over sessions is misguided and overly simplistic. Context always matters.

edit: formatting

Please tell us how the following assumes that authorization data and user data lives in the same database:

"You wanted to implement log-out, so now you’re keeping an allowlist of valid JWTs, or a denylist of revoked JWTs. To check this you hit the database on each request."

If authorization data is stored outside my database, I may not store anything related to that user other than their unique ID. I may not know anything about the JWTs because a middleware talking to the external authorization service authorizes a request before I ever query my database.

I could also store valid JWTs in my own database or in a caching layer, say with redis for quick access. There's nothing wrong with that, but without more details on what someone is building and what it needs to be designed for there is no absolute right or wrong answer.

How you organize your data, your services, your middleware, etc. does not change this the slightest, hence are irrelevant.

----

[1] proper logout is instant (this invalidates the "just use a JWT with short timeout!" argument), and prevents session stealing (this invalidates the "just throw away your JWT to logout" argument)

Isn't that kind of the point though? I'm not saying JWTs are a better answer for logout, only that JWTs can still support logout and they both similarly require a database transaction.

If logout can work either way, and in all likelihood logout isn't a performance concern as long as its secure, you may have other use cases and requirements in an app that do make JWTs a better fit.

I still don't understand how the argument here can be that there are no cases for JWTs, or that the list of "its" in the original article can be bulletproof no matter what else is going on in a project. That's unnecessarily absolute and leaves no wiggle room for competing priorities and constraints.

Because the whole selling point of JWTs was that they eliminate this database transaction!!!

As soon as you have logout, your JWTs degrade to nothing but a unique byte sequence that you need to check in the DB whether it's still valid or not. You may as well use unique JPEGs or MP3 files to identify your sessions - it would work just as well (and it would be equally stupid).

In fact, in the presence of logout, JWTs are worse than opaque session tokens, because careless programmers developing a new microservice might see that it's a JWT and be tempted to just check the signature and the expiration without consulting the auth database for revocations, thus leaving that microservice accessible with compromised tokens.

For authentication and authorization there isn't much benefit, but that doesn't make one right and one wrong. They can be useful in certain author scenarios, like if you're using an external author service.

If the argument here is simply that JWTs aren't a huge benefit in auth, sure that's fine. I just don't agree that that means they should never be used for auth or that they are universally worse for auth in all scenarios and applications.

What you do to authenticate between the BFF (for lack of a better name) and other services is a different matter.

There is a huuuuuge gap between services handling 10k req/s and Google/Facebook.

I think one big upside with JWT that he doesn't mention is that if you have some services geographically distributed, then having decentralized auth with JWTs is quite nice without having to geographically distribute you auth backend system.

So, yes, if you have a monolith or services colocated, or have some kind of monolothic API layer, then no, perhaps JWT does not make sense. But for a lot of distributed services, having JWTs makes perfect sense.

And you don't have to introduce JWT revocation for logout, if you have short token expirations, you can accept the risk of token leakage. If the token is valid for like 30 seconds or 1 minute, you would probably never be able to notice that a token has been leaked anyway.

There is no reason you can’t keep a list of valid “session” identifiers and check the JWT is valid against that as part of verification. Then the only state you need to store server-side is the identifier. You get the exact same benefits of server-based session stores without needing to store the entire session on the server - just the identifier.

JWT makes it possible to distribute the same access token across multiple systems, but so do stateful tokens. The security implications when you're using JWT for this solution are much higher than with database tokens. Let's look at this for a moment:

JWT Security Issues:

- Inherent design issues (alg=none, algorithm confusion, weak ciphers like RSA)

- Cannot be revoked.

- Key rotation and distribution is necessary to keep token safe over a long period of time

- Claim parsing needs to be standardized and enforced correctly in all services (otherwise impersonation is possible)

Database Tokens Security Issues:

- Timing Attacks against B-Tree indexes [1]

- Giving direct access to the database to all microservices is risky

The security issues with databases are ridiculously easy to solve: To prevent timing attacks, you can just use a hash table index, split the token into a search-key part and a constant-time-compare part or add an HMAC to your token. To prevent direct access to the database by all your microservices, you just wrap it with an API the verifies tokens.

The JWT security issues are much harder to solve. To prevent misconfiguration or misuse and standardize the way claims are used across your organization, you probably need to write your own library or deploy an API gateway. To counter the lack of revocation support, you either need to use very short lived access tokens (so your refresh token DB will still get a lot of hits and you would still need to deal with all the scaling issues) or set up a distributed revocation service (not easy at all). Setting up seamless key rotation also requires additional infrastructure that is not part of the hundreds of JWT libraries out there.

It's really easy to get a JWT solution that just works and scales easily, but if you really care about security — especially if you care about security! — JWTs are not necessarily easier than stateful tokens. They're probably harder.

> Apart from that, Google and Facebook don't even use JWTs between the browser and backends after the initial login but actually do have some sort of distributed session concept last time I checked.

Last time I checked (which was today for Google), neither Google, nor Facebook is using JWT for their access or refresh tokens. The only place I saw JWT with the ID Token in their Open ID Connect Flow, and they can't really avoid that even if their wanted, since this is mandated by the spec.

Facebook and Google don't need JWT. Scaling and distributing a read-only token database to handle a large amount of traffic is easier — not harder! — for these companies. Stateless tokens can be useful for them in certain scenarios, but even then, if you're at Google or Facebook's scale, why would you opt for JWT over an in-house format that is smaller, faster and suffers from less vulnerabilities?

[1] https://www.usenix.org/legacy/event/woot07/tech/full_papers/...

Even with asymmetric tokens and a key distribution URL, you still have to make sure the clients periodically update their list of keys — this is not something you get built-in with every JWT library. And you still have to setup a mechanism for generating the keys and distributing them between the various instances of your auth server. This is not so hard nowadays with cloud KMS services, but setting up this solution on our own infra was quite painful.

> I agree that a lot can go wrong, but isn't that also true for home growing a distributed database tokens solution (I surely have seen some monsters in the wild). So can't the problems with both solutions be mitigated using some good libraries?

My point is not that a database solution is without its own issues. At my $DayJob we're also using a mix of stateful and stateless tokens (with distributed revocation lists) and we had to deal with issues with both of them. But stateful tokens on a database rarely suffer from security issues — the issues we had were always related to scalability and performance. The mitigations for these issues are also different: they almost always have to do with optimizing the infrastructure (scaling out the database, adding a cache) rather than using a library. In fact, when we use stateful tokens in a distributed scenario (and I'm sure this is true for almost everyone out there), all token handling is centralized in the auth service, so libraries are not strictly necessary. At most, client libraries would be very thin wrapper around HTTP API calls.

This post doesn't seem to address microservice architectures at all? For me, this is the primary reason to use JWT's -- so you can pass authentication ("claims", or whatever you want to call them) through your chain of microservice service-to-service calls. If you don't have microservices then there's much less reason to use JWT's.

I'm not saying the article is a strawman exactly, but it does seem to miss the primary use case of JWT's. At least, the way I've used them in anger.

Also, the "JWT's can be insecure if you use the wrong library or configure them incorrectly" argument, while having some points, seems to me more of an argument that you should really do due diligence on any libraries you use for security. The better JWT libraries are not insecure by default.

I wouldn't use JWT's if I were making a monolith, but there are lots of companies who (for better or worse) use microservices.

This is a misconception about so called zero trust. You can't "just" pass the same token to someone else. They can use it to impersonate or misuse the token later. While you are going to say that "my microservices will not impersonate users because to each other, they are all trusted," you have run directly into the difference between trusted and zero trust.

This is why JWTs are used in OIDC (e.g. “Sign-in with Google”: any website can use it, and it doesn’t make Google’s own security weaker.

I’ll concede that small, but important, details like these are not readily understood by those following some tutorial off some coding-camp content-farm (or worse: using a shared-secret for signing tokens instead of asymmetric cryptography, ugh) - and that’s also where we see the vulnerabilities. OAuth2+OIDC is very hard to grok.

What’s so terrible about that? Several security engineers I trust favor designs with symmetric crypto, e.g. Fly.io https://fly.io/blog/macaroons-escalated-quickly/ and Facebook https://eprint.iacr.org/2018/413.pdf

My interpretation is that the audience and scope claims, as other features like nonce, are in place to prevent tokens from being intercepted and misused, not to facilitate passing tokens around.

The purpose of a nonce is to explicitly prevent the token from being reused.

The purpose of the other claims is to prevent them from being accepted (and used) in calls to other services.

If you implement your server correctly, each instance of each service is a principal which goes through auth flows independently and uses its own tokens.

There is no token sharing.

[1] https://salt.security/blog/oh-auth-abusing-oauth-to-take-ove...

Is it good? I'm not a fan. To use DPoP safely (without replay attacks), you need to add server-side nonces ("nonce") and client-generated nonces ("jti", great and definitely not confusing terminology there).

You need to make sure client-generated nonces are only used once, which requires setting up... wait for it... A database! And if you'll be using DPoP in a distributed manner, with access tokens then, well, a database shared across all services. And this is not an easy-to-scale read-oriented database like you'd have to use for stateful tokens. No, this is a database that requires an equal number of reads and writes (assuming you're not under a DDoS attack): for each DPoP validation, you'd need to read the nonce and then add it to the database. You'd also need to implement some sort of TTL mechanism to prevent the database from growing forever and implement strong rate limitation across all services to prevent very easy DDoS.

It seems like the main driving motivation behind DPoP is to mitigate the cost of refresh tokens being exfiltrated from public clients using XSS attacks, but I believe it is too cumbersome to be used securely as a general mechanism for safe token delegation that prevents "pass-the-token" attacks.

[1] https://developer.okta.com/docs/guides/dpop/nonoktaresources...

Proof tokens can only be used for a narrow window of time (seconds to minutes), so you just need a cache of recently seen token identifiers (jtis) to do replay detection. And proof tokens are bound to an endpoint with the htm and htu claims. They can't be used across services, so I don't see a need for that replay cache to be shared across all services.

Please note that I am talking about using DPoP to verify _every_ request, not just a token refresh request (where OAuth 2.1 is setting DPoP as an alternative to issuing a new refresh token and revoking the old one). When using DPoP for every request, the amount of client-generated nonces ("jti"s) is quite high, since you need a new one for every request.

And yes, you can rely on "htu" to distinguish between services and have a separate nonce cache for every service, but this would require deploying and maintaining additional infrastructure for every service. Depending on your organization this may or may not be an issue, but this is a big issue for us.

What did we decide on instead? Request Signature and Mutual TLS binding (RFC 8705) where possible. Request Signatures without nonces do not work well for repeatable requests (like the Refresh Token Grant), but this is not our use case.

Put another way, JWTs used as bearer tokens have vulnerable to intra-audience replay attacks.

While this is true for many zero trust architectures, but you don't have to build zero trust architectures this way. Simply have the token commit to a public key of a signing key held by the identity, then you can do Proof-of-Possession and remove these replay attacks. This is the direction zero trust is headed. For instance AWS is slowly moving toward this with sigV4A. Most zero trust solutions aren't there yet.

It's right in the name.

Anyway, zero trust architecture are wildly overrated and used in way more places than they should. But the entire thread is correct in that you can't build them with bearer tokens.

1. The client that holds the token can't use an HSM or SSM to protect the token because they need to transmit it. Thus a compromise of the client via an XSS or Malware, results in the token leaking.

2. The server that receives the token, might be compromised and they can replay the token to other servers or leak it accidentally e.g., with a log file or to an analytics service.

Both of these problems go away if you uses OpenPubkey or Verifiable Credentials with JWTs. The JWT is now a public value, and the client holds a signing key.

1. The client can protect the signing key with an HSM or SSM (modern web browsers grant javascript access to a SSM).

2. The server only receives the JWT (now a public value) and a signature specific to that server. They don't have any secrets to protect.

> But the entire thread is correct in that you can't build them with bearer tokens.

You can and people do, but it is far better to use proof of possession JWTs than bearer JWTs. Even better to use JWS instead of JWTs so you can make use of multiple JWS signers (a JWT is a type of JWS, but a JWS with more than one signer can not be a JWT).

What types of tokens do you suggest in each case?

1. Alice generates an ephemeral key pair (if she is using a browser she can generate the key pair as a "non-extractable key"[1]).

2. Alice gets ID Token issued by Google that commits to their public key,

3. Alice signs her API request data to Service A and sends her ID Token to Service A.

4. Service A checks the ID Token (JWT) is issued by Google and that the identity (alice@gmail.com) is authorized to make this API call, then it extracts Alice's public key from the ID Token and verifies the signature on the data in the API call. Then it passes the signed data to Service B.

5. Service B verifies everything again including that the data is validly signed by Alice. Service B could then write this data and its cryptographic prominence into the database.

Technically OpenPubkey uses a JWS, but it is a JWS composed of a JWT (ID Token) with additional signatures. OpenPubkey signed messages, like the ones passed via the API are also JWS.

I'm working on a system where each service in the path adds their signatures to the signed message so you can cryptographically enforce that messages must pass through particular services and then check that at during the database write or read. Using signature aggregation, you don't get a linear increase in verification cost as the number of signatures increase. It doesn't seem to add much overhead to service meshes since they are already standing up and tearing down mTLS tunnels.

The main question to me is how much autonomy do you want to give to your services. There are cases in which you want services to query each other without those services having to prove that the call originated from a specific authorized user.

[0]: https://github.com/openpubkey/openpubkey

[1]: https://developer.mozilla.org/en-US/docs/Web/API/CryptoKey/e...

Really the ergonomics are why this hasn't been adopted more readily. I wonder why it's possible to have OpenTelemetry inject a header into thousands of different APIs and services for dozens of programming languages, more or less flawlessly. But if I wanted to do this at process boundaries, and the content of my header was the result of a stateless function of the current value of the header (aka token exchange + destination service): you are shit out of luck. Ultimately platform providers like Google, Apple and Meta lose their power when people do this, so I feel like the most sophisticated and cranky agitators are more or less right that the user experience is subordinate to the S&P top 10's bottom line, not real security concerns.

> I don't really want Google or Apple or whoever has, for idiosyncratic reasons, possession of a private key, to sign every request I make to Google Calendar, because they will inevitably revoke it sooner for obnoxious business reasons than any good security reason.

Can you provide more context on this? I would assume asymmetric signing keys are less likely to be revoked than say an HMAC key since an HMAC key must be securely stored at both the client and server whereas you can just put a asymmetric signing key in an HSM at the client and be done with it.

Fair point. This post assumes a single database which opaque tokens can be mapped to. That said, a lot of smaller webapps are and should be monoliths.

Is it about battlefield and such?

My understanding: To use something "for real" on an actual project, not just toy around with it. (What you use can be good or bad, expression doesn't say)

For what it's worth[0]:

> If you do something in anger, you do it in a real or important situation as it is intended to be done, rather than just learning or hearing about it

[0] https://dictionary.cambridge.org/dictionary/english/in-anger

Or just, you know, service architectures. Most microservice architectures I've seen go way too far down the route of breaking up services and their infrastructure to an impractical level. But all you need in order to make JWTs really useful is two federated services. This happens all the time, often in the course of some partnership, acquisition, or just an organizational structure meant to decouple 2+ teams from each other.

Based on their mention of a single framework connecting to a single database, OP seems to have never moved past the point of developing a single service. Which is fine! It makes things simpler for sure, and you can get very far with that. But they are then dispensing advice about things they don't seem to know much about.

I disagree with the whole "you're not Google/FB"/"over arbitary RPS" logic though. If the design makes sense then it makes sense, end of story. Just understand it.. lol

Not needing a DB connection to verify means you don't need to plumb a DB credentials or identity based auth into your service - simple.

Being able to decode it to see its contents really aids debugging, you don't need to look in the DB - simple.

If you have a lot of individual services which share the same auth system, you can manage logins into multiple apps and API's really easily.

That article seems to dislike JWT's, but they're just a tool. You can use them in a simple way that's good enough for you, or you can overengineer a JWT based authentication mechanism, in which case they're terrible. Whether or not to use them doesn't really depend on their nature, but rather, your approach.

It's not difficult to implement JWT's, the concept is simple, however, with authentication code, the devil is in the details, and that's true for any approach, whether it's JWT's, or opaque API tokens, whatever. There are many, many ways to make a mistake which allows a bypass. Simple concepts can have complex implementations. A JWT is simply a bit of JSON that's been signed by someone that you trust. There are many ways to get that wrong!

Convenience, when it comes to auth, is also usually the best path, and you need to be careful to use well known and well tested libraries.

"You don't need all of that!" sure but you probably already have it somewhere in your codebase and it's pretty universal. You also probably don't utilize every feature of http itself, that isn't a cogent argument against using http.

JWTs are supported by a large number of tools, libraries, middleware appliances, etc; there's a huge ecosystem out there to support it.

You also might delegate auth to a third party like Auth0 or FusionAuth so that you don't handle any PII, because all of the PII is handled by a vendor, and you only store application-specific data.

"You want to implement logout" means a few things; in most apps you just ... have the client forget the token and you go about your day and it's fine. "but what about if a nefarious actor stole the token!!!" you might say, but hand-rolled session tokens have the same problem.

"You want to turn off access for all users" is something you can do in http middleware; e.g., I have used middleware that do things like "only allow through requests that have jwts with the 'admin' role in their claims because we have turned off the system from users for downtime", and that works fine. (specifically I wrote a traefik plugin to do this in an afternoon).

"You want to ban a single specific user really quickly" is a thing JWT won't do out of box.

If you read this and think "this doesn't apply as long as I have to use JWTs for some service I rely on, anyways", you missed its point.

I think there's a cyclic thing that's been happening for years where people in the security community like to talk about how bad jwt is, but then not produce anything that meets application developers needs in a meaningful way. I spent years avoiding jwt and ultimately found avoiding jwts wasn't actually a good use of my time.

> JWTs are motivated by migrating authN into its own independently-scaled microservice

that's definitely one use-case, although I don't actually think having auth in a separate microservice under your own custody is the dominant use-case. The auth being in an entirely separate database means PII is in an entirely separate database, which can be a useful access control mechanism, or in the case of using a third-party auth service, means that there is no PII in databases under your custody at all. It makes the situation of "developers can access all data created by the software that they work on" really easy to implement while also maintaining "developers do not have access to all of the PII" as an access barrier. I think "I just use Auth0/FusionAuth and don't think about it" is actually the dominant use-case, and every third party offering that kind of developer experience utilizes stateless session tokens to make that happen.

> If you just use your framework session system, you don't ever think about refresh tokens.

right, but now the problem of keeping PII data access rules separate from your application domain data is an additional thing you have to engineer and think about securing, so I think the article is underestimating some of the negatives of that tradeoff; I've never seen a framework with a built in session system that did a good job of keeping access to the PII separate from the application data.

But I don't want to pretend we're still having the same conversation that started up thread. I assume you take my point, that if you think "I already have to do JWT so I don't save anything by not using them everywhere" rebuts the post, you've misread it a bit.

I have never believed that and I don't think my comment ever suggested I did, I mentioned "you might have jwt-handling tooling already" as one concern among many, not an entire argument. It really seems like you've honed in on a single point and, as you would say, misapprehended my comment. That one argument is not reason alone to use JWT and I really don't think my original comment ever implied I thought it was the entirety of the topic.

We’re rebuilding our microservice hell as a monolith anyway.

I see some strong arguments for standardizing on a single one.

1) You may not want your application servers having direct access to your auth service or auth database. You may not have the resources to control employee access to sensitive data when it’s shared across services. You may want to spend limited resources for security audits on the systems that contain the most sensitive data, and having them separated from everything else is helpful.

2) Depending on what 3rd party services you use it may not even be practical to have connectivity between auth and other services, and if you do, the latency may be bad enough that you wouldn’t want it to be blocking every request. This is especially compelling for hybrid environments, e.g. the 20 year old database in a colo with your user data, and a new service being built for you by consultants on a PaaS.

3) People act like revocation is such a nightmare, but you only need the auth service to sign invalidation tokens to be passed to the client-facing services, and those services only need to retain them for the max TTL of the auth tokens, after which they can be evicted. Yes, that’s something that could be motivated by having a massive environment like Google, it could also just be a way to keep costs down when you’re paying by the byte of storage or by the outbound request on some cloud. You could try to make the argument that storing invalidation data is just as bad as storing session data, but the key questions are “Where?” and “For how long?”, and then in some circumstances it breaks down very quickly.

4) You may not want sensitive user data stored in the jurisdictions where you want to host your apps. That’s not a big company problem, it’s dependent on the nature of the services you provide.

I keep seeing the idea that JWT was designed for Google or Facebook scale being repeated over and over, but the reality is that neither company uses it. Last time I checked, both used rather short access and refresh tokens and it appears that at least the refresh token (if not both) is stateful.

Implementing stateful tokens on a global scale and sharing them across hundreds of service is HARD, but it's easier when you are Google or Facebook, and you've got enough resources to throw at this trouble.

And if you do need to implement a stateful token, you've got every reason to choose your own format. Your applications are using your own authentication libraries and infrastructure (e.g. API gateways), so you don't have to worry about complicating their life with a non-standard format. The upside for using your own stateless format is that you avoid all the design issues with JWT (alg=none, algorithm confusion, questionable support for outdated algorithms from the 1970s) and you can design a far more compact format that takes a fraction of the size of JWT[1].

There's a reason JWT got popular with scrappy startups, enterprises hobby projects and Udemy/Medium tutorial authors: they're very easy to spin up and library support is everywhere. I don't mean to say JWT is the right choice for any of these uses — it probably isn't. But it's the easy choice, the worse-is-better solution. The worse solution needs only be better in one respect to win: it should be easier to implement, copy and spread.

In the end of the day, JWT is not a good solution for either Google-scale companies or small startups. But it's the small startups that usually lack the resources and awareness to adopt another solution.

[1] https://fly.io/blog/api-tokens-a-tedious-survey/#protobuf

Not only is this true, but most actual deployments of JWTs just have you swap a JWT (ID Token) for a opaque session token.

That said, I really like having a JWT signed by an IDP which states the user's identity because if designed correctly you only need to trust one party IDP. For instance Google (the IDP) is the ideal party to identify a gmail email address since you already have to trust them for this. I created OpenPubkey to leverage JWTs, while minimizing and in some cases removing trust.

OpenPubkey[0, 1] let's you turn JWTs bearer tokens into certificates. This lets you use digital signatures with ephemeral secrets.

[0]: https://github.com/openpubkey/openpubkey [1]: https://eprint.iacr.org/2023/296

This looks really awesome, thanks for sharing.

To be able to sign a message you not only need the ID token but also the private/signing key, and the corresponding public key is bound to the ID token (using the nonce field).

Thus you can prove that not only did Google say you are you, but you possess the signing key associated with the ID token that says so. Thus I can be sure someone else didn't just steal your token in flight or from a log file for example.

JWT bearer tokens use a signature to issue an identity, but that don't include the public key of that identity. The issuer has a public key, but the issuee does not.

There are plenty of JWTs that are certificates:

* proof-of-possession JWTs,

* self-issued JWTs, etc...

1. It standardizes your auth system. While sessions auth is mostly implemented in the same way across systems, learning oauth and oidc gives you a standard across the industry.

2. Jwts give an easy path to make “front end” applications and api authentication work in the same way. This in theory reduces your security surface area as all of your authnz code can be shared across your offerings.

If a short session time isnt good enough, you can use a simple key store to check for revoked tokens. Youll be hitting a db but its somewhat better since its just a very small db of revoked tokens.

Its hard for me to imagine though with like a 30 min or even few hour long token, under what circumstances you'd actually revoke tokens. If your db got leaked, you can rotate the key and invalidate all tokens. Otherwise, itd have to be something like you have some post login fraud detection in place. Cause jwt or not, if a user just signed in and a hacker got their auth token, what are you going to do? Sure you need to check the db to revoke it, but the problem is how would you know the tokens been compromised?

It's not bad solution per se, but it does negate JWT's main value proposition, which is to not need such a store.

Maybe I'm damaged from working in a regulated industry, but a (possibly malicious) client who went through the logout process and could prove someone else reused the token after logout might have a case. Or another of a myriad of unknown possibilities.

It's all very unnecessary. There's a reason we all used to invalidate trust server side. It's just so much easier to reason about.

This thread is really making me despair. If you don't see a problem with JWTs, you aren't experienced enough to use JWTs.

If an attacker is able to steal a victim's cookie database, their system (or at the very least, their browser) is already deeply compromised. It is very likely that an attacker with such capabilities could prevent your website from ever sending the logout request (install a browser extension which blocks it, inject into the render process to silently drop it, modify the cached JavaScript on disk to inject code into the site, etc.). The logout functionality only works insofar as you trust the client, and in any circumstances where the client's cookies could be stolen you really can't trust the client. So logout revocation is not really a meaningful security boundary.

This can't be done without revocation.

You would need revocation in this case. Implementing this is easier when you already have database backed tokens, but unless you intend to support these features JWT is a totally reasonable engineering decision.

You have to store invalidated tokens anywhere they might pass through a service, which means you have to persist them for as long you can predict that there expiry will last. Simply putting them in a memory database isn't 100% if that db gets flushed, and then you might start storing them in a disk database, which at that point, you might as well have just read the db in the first place using cookie auth.

In microservices, you generally have to put an invalidated JWT cache between every service, or compromised JWT's are just floating around your intranet.

I've worked at a plethora of places who have JWT's who have no invalidation strategy what so ever, the majority of developers think that when you log out and the user has forgotten the JWT then you are all good......

Or Do all the server instances store a copy of all revoked tokens in memory/local db?

That keeps your application DB free for application load, while keeping your identity validation logic nice and snappy.

Of course, adding infrastructure may be intimidating, but most applications that face any real load are going to be using redis or something similar anyway at some place in the stack.

The secondary advantage is you don't need to deal with cookie storage, sticky sessions or anything else along those lines.

If you're manually hand crafting a server, go for it. If you're treating them like cattle not pets, going stateless with a bearer token tends to be easier.

I don't understand this argument against JWTs; if instantaneous session revocation is important for your use-case, versus JWTs more typical 5 minute or 60 minute expiration, there is nothing about JWTs which makes them poor candidates for going to the database as you would a session token. And, you get all the other benefits.

One example where this can matter: I've seen JWTs used in defense-in-depth scenarios where you've got an API gateway that does the initial JWT validation, including a round-trip to the database to check for revocation, but then had microservices behind the gateway only check the signature. Traditional session keys would require a database roundtrip for every validation, which could number in the dozens for a single API request.

You would be required to keep a Map<UserId, Timestamp> in memory, potentially with TTL. Most systems can handle this easily for their expected user load. If not, you should have the engineering capacity to figure it out ;)

Logout button sets reset_time to now, as does revoking tokens. This would only allow you to revoke all tokens for one user at the same time, but this tends to be fine, since JWTs should be short-lived anyway and apps should deal with the expectation of them being expired/revoked.

Even if the TTL is short, there are plenty of ways to compromise a token and use it immediately in an automated system.

If you're using JWTs, I'd lean shorter TTLs and embrace this as a potential concern. Not sure what the best re-auth frequency is though. I'd be really interested to see other's thoughts on that.

Yet we have millions of passwords in dumps across the internet. Maybe hijacking the client or server is more common than thought?

Passwords being leaked is due to noobs and idiots in charge of systems, and probably not using an actual auth provider.

It really isn't that hard.

1. Send secure information over SSL. If some noob-tier dev decides that's too advanced for them, congrats I guess for being stupid.

2. Store passwords with hashing and salting. If you use an auth provider, like auth0 or firebase even, they will do this for you because they're not noobs. A noob-tier dev who stores plaintext passwords in a database with insecure connection and postgres:postgres is again stupid af.

Most of those leaked passwords are because of this. And don't think some Fortune 500 is not stupid. They outsource their development to Accenture or Detoilette, or any of the other outsourcers where they pay some "Software Architect" $8/hr to secure banking information (while charging the client $150/hr). I'm not throwing shade, but these companies are cheap af and throw bodies at the problem instead of experience and brains. I have direct experience with this, so I know how bad it is.

Don't confuse people being stupid (very common, 90% of any population) with a failure in a technology that others who don't understand it claim makes it inferior or something.

The internet and all the technology you use today is glued together by the <10% of people who actually understand computers, networking, and problem solving.

Just because a noob-dev uses session tokens doesn't mean it will be any secure if they fail those 2 points I mentioned. Those are so deadass simple to mitigate, anyone who purposefully skips on them should be named and shamed, and receive a 10 year cooldown from being a software developer.

I have a template that's backed by terraform and the authentication client is in lambda so the whole thing is serverless, self-contained and practically free. So I just run "terraform apply" and I have scalable auth for my new service.

https://github.com/alshdavid/template-cognito (only 1 dependency on AWS, everything else is stdlib)

If any service I create is lucky enough to break out of the free-tier and cost is an issue, then I can just move to another OAuth2/OIDC provider. The auth mechanism Cognito uses is just a specification meaning I am not coupled to any one service provider (though the user accounts themselves are). Cognito, Auth0, IdentifyServer, or whatever - I can migrate if cost becomes a problem.

The big issue with JWTs are that, if lost, they give permissions to attackers without revocability.

For this reason, I keep auth-tokens short lived and refresh them often. Refresh-tokens are revocable and live for a few days. This means that a lost auth-token is only harmful for a few minutes while a lost refresh token is only harmful until revoked or expired.

Tokens are stored as path-specific http-only cookies so the only vector for attack is if a user physically opens devtools and gives an attacker the token - or if the attacker has access to the computer (physically or via a malicious terminal script).

High risk operations (e.g. delete account, delete content, anything high risk) requires "step-up" authentication - so a user is asked to re-authenticate in those cases.

Overall, when you consider that rolling your own authentication comes with the liability associated with holding user data (companies must announce a breach to users, etc) - if a service provider like Cognito is compromised, you won't be liable or the only one affected.

JWTs have security concerns, but on balance, when used with third party provider, a sensible configuration and considering the risk of rolling your own - they are fine.

> absolutely no one who is not Google/Facebook needs to put up with the ensuing tradeoffs. If you process less than 10k requests per second, you’re not Google nor are you Facebook

What's the magic property that flips when you pass 10K requests per second? Are we sure it's at 10K requests per second, not 8K? or 5K? In general, at that kind of scale I'd think JWTs would become less appealing - AWS operates on IAM for example.

And why are Google and Facebook the best examples of companies who are operating at scale? There are different kinds of scale than just 'ad auctions per second'. I would imagine the access management concerns of, say, JP Morgan Chase are at least as complex and challenging to scale as those of Facebook.

Making authentication only require a signature verification at the edge (JWT) vs authentication middleware that needs to do a DB read (opaque), can be a life saver even if you have 10 requests a second most of the time.

Q: What about the endpoint that's issuing the tokens?

If you are issuing the tokens yourself, you can build a simple horizontally scaling identity service that only does authentication and token issuance. With refresh tokens, if that service goes down it only prevents users not already signed in from signing in. Generally users stay signed into to webapps for weeks at a time, so you have massively reduced the impact: rather than 100% of your users not being able to do anything on your site, now 0.5% of users are impacted.

There are definitely arguments to me made against ridiculous over-engineering, modern web dev has taken the problems of 1% of engineers and made them problems for 100% of engineers, but I think this is a bit of a silly one to focus on

A centralized database for sessions, and we can extend/revoke any token anytime.

I'd argue that people who need to avoid hitting their database on every request outnumber people who need sub-minute revocation

More so, A revocation list on something like Redis can expire with a token and a lot less expensive than an rdbms lookup with joins.

Not too mention, why add extra load with extra DB calls in the first place?

You don't have to be Google scale to want to separate logins from the main service. And that's just for starters.

The real thing to push back on is the logout requirement. Everyone pretends they need this, when what almost everyone should do is just mandate appropriately short token lifetimes and revoke refresh tokens as needed.

That’s a logout requirement?

Blocking refresh in the authorization server is trivial, but trying to implement the same on access tokens in the resource server at the point of use breaks the entire security model of JWT. It's unreliable, because now every resource server has to take on partial responsibility for authorization which multiplies opportunities for mistakes. As the OP points out, you need to keep track of some sort of block list and lose out on many of the benefits of JWT (i.e. a resource server being able to rely fully on claims in a signed token before allowing an action).

When people show up with this kind of requirement, in my experience, it is often because they foolishly configured a client with a very long expiration on access tokens (e.g. ~months/years instead of ~minutes/hours). This creates a problem when some aspect of a user's access needs to change (e.g. disgruntled employee was fired, customer didn't pay their bill, etc). You can address this more easily by pairing a short access token lifetime with a long refresh token lifetime.

Additionally, when using asymmetric validation, you can rely on JWTs as licenses: Your software can restrict offline features based on a locally stored token, simply by checking that the JWT was signed by the authority. In tandem with the ability to store metadata, your app (with code held by the untrusted user) can use the token to determine the user's license features without requiring an always on connection. (Obviously patching out the license checks is another matter)

These features can be layered on top of opaque tokens, but since a JWT has all the benefits of an opaque token (store the opaque token as a claim just like the rest of the metadata), it's actually a complete package that does it all without needing to roll it yourself.

Revocation is just hard. Even PKI's CRLs/OCSP don't have a subscription mechanism by which an end-point can get revocations of some small set of certificates. Even a public feed of revocations could be problematic if you're going to terminate employees because the timing has to be perfect: too early and you're tipping off employees, too late and you're allowing disgruntled employees run malware on your systems and network. In practice being late to revoke access is generally accepted as acceptable as long as it's not too late.

I don't think I would start with JWT if I did this again.

Yes, query the DB or some sort of storage every time. It sounded so clean and nice and fast to just check JWTs without any network calls. But it ended up very messy and complicated. Might still be worth it in some cases, of course, but I would start my next project with random sessions stored in a db or redis or memcache or .. something :)

You can actually do crazy stuff with your sessions as well, to avoid a normal db lookup. But in practice all services I have worked on would/did not suffer noticeably for a fast DB lookup.

I've experienced more than once, issues where the auth service has bugs and the logged out session is still valid for a long time. Or an attacker that figured out the micro services blueprint and now had authed access to the entire network.

Jwt is still useful between services, however the front end can do just fine with a session id that can be easily revoked.

In my projects, we have used encrypted JWTs and it seems to me a fine solution. Log out can be implemented in a user facing client by deleting the JWT and refresh token. Given a short enough expiration time, this is sufficient for most use cases involving user facing applications. Isn’t it? Generally, at least in the domains of the applications I’ve worked on, users only intend to log out of their current application when logging out. Meaning if they are signed in on their phone and on their desktop browser, when they sign out in the browser they don’t intend to also log out of their phone application.

The only downside I see is that if you want to log out of all sessions it is impossible to implement without maintaining session state in the server.

JWT's security -and that of all bearer token systems- depends utterly on the use of TLS.

The security of JWT used correctly is comparable to that of Kerberos or PKI. There are some things to look out for, like that it's way way too easy to write implementations that fail open (e.g., you don't recognized a signature algorithm so you don't validate the signature but also fail to fail), but other than that there's nothing particularly bad about JWT.

> Log out can be implemented in a user facing client by deleting the JWT and refresh token.

Logout has to be implemented either by letting time pass w/o re-authenticating (i.e., let the session expire) or telling the server that you're logging out. You can login again, even with the same token as before (IFF it's not expired yet). There is no reason to have to "burn" a token, and that would add a significant amount of complexity akin to revocation.

> The only downside I see is that if you want to log out of all sessions it is impossible to implement without maintaining session state in the server.

Sessions imply session state. What you probably meant is that if you want to revoke access then you need a revocation system, and that effectively bolts on extra state on your servers (and a pub/sub system). Generally no one wants that headache.

No they don’t. JWTs can be used with third party services that should never see a refresh token. Those are the requests that should be presumed vulnerable.

Suppose you use Azure. You may have one or more app registrations with defined roles as well as apis exposed within the Azure config. It's convenient to acquire an access token that can be sent to various apis, each of which can accept the token without having to worry about session state or really anything to do with authentication other than validating the token's legitimacy. If I wanted to roll my own security, maybe JWT isn't how I would choose to do it, but I definitely don't want to do that.

For the record, I use JWTs, I have hardly any scale at all. I have none of the requirements that would hit the database on each request. It works just fine and I didn't need to consult an article to tell me what to think because, and this is the crazy part I know, I'm able to break apart my business requirements and decide for myself what is needed (or not). JWTs work just fine in my case. YMMV.

The security arch is mainly in web/auth and web/interceptor packages if anyone is interested in learning from the code. It uses connectrpc, which has a nice interceptor arch.

Happy to share Vera’s thesis report if anyone is interested…

Please share her thesis. I'm interested.

Everyone wants to be big. You can either frontload those problems, and risk drowning in irrelevant complexity, or defer them. The problem with deferring is that switching lanes appears like it will be expensive and time consuming.

The secret is that if you solve actual problems, rather than dreams and future hopes, the answers are almost always obvious. The work may be painful and people may be angry, but the rationale will be clear.

You need to be able to turn away malicious users as fast as possible. If you take the time to check a database first, that's a precious resource they can consume.

"Add a cache!", you might say? What if they use random client id and client secret for every request, how do you cache against that?

At some point the attacker either gives us or just switches to the brute force of flooding the pipes with so much traffic that the AS doing the filtering goes down. At that point it is now someone else's problem. They might start de-peering the ASes generating the traffic.