`nix-shell` (that is what the OP seems to be referring) is always slow-ish (not really that slow if you are used with e.g.: Java CLI commands, but definitely slower than I would like) because it doesn't cache evaluations AFAIK.

Flakes has caching but support for `nix shell` as shebang is relatively new (nix 2.19) and not widespread.