Sub-µm accuracy is not at all easy for a CNC mill and the surface quality is way to bad for optics.

Commercial aspheres are polished with special CNC grinders, but achieve nowhere near the quality of spherical lenses. Only recently magnetorheological finishing became available which allows getting a bit closer in quality.

In most applications, it makes more sense to stack a few spherical lenses instead of manufacturing a super expensive custom asphere which cannot even compensate for chromatic aberrations.

Doable? A common turning precision (you don't want to mill figures of rotation) is 0.01mm or so. You need four more orders of magnitude.

I'd love to read up on that not so specialist CNC tech, considering most of the CNCs brag about having tolerances of ± 0.0025mm.

Only if you consider a Moore Nanotech diamond-turning machine "not so specialist"...