It is not like it is not tried. Wavelet compression never took off. I don't know if it is because the format of is just better, or there was never enough investment into those formats.
The problem with Wavelet based compression was
since wavelet transforms were applied globally
to the whole frame at a time,
while they were suitable for still image compression
they couldn't really take advantage of motion compensation
so their applicability for video was low.
Same with fractal based techniques.
Besides, as resolutions got higher and higher
the blockiness of the 8x8 DCT became less and less a factor
Digital cinema uses wavelet compression - intra-frame only JPEG2000 at hundreds of Mbps. It seems that at high resolution and bitrate it actually performs similarly to or better than h264, e.g. this paper and its references: http://alumni.media.mit.edu/~shiboxin/files/Shi_ICME08.pdf
Digital cinema uses a resolution that is much higher than H.265's targeted sweet spot.
Their quality needs are also a lot higher.
Motion compensated video cannot give them the desired quality.
Hence intra frame only wavelet based compression.
Also, note that JPEG2000 which uses wavelets implies that for still images, wavelets can be made to work better.
JPEG which preceded JPEG2000 was 8x8 DCT based.
You can still use wavelet compression to encode the residual from MC, but I think the biggest problem is performance: DCTs have been optimised far more than wavelet transforms.
Even in still-image compression, the difference is noticeable --- I have some high-resolution PDFs containing JPEG2000 scanned images, and they take significantly longer to render than the equivalent containing JPEG images.
Well, you could.
But the way current schemes are structured,
the motion compensation is done on a 16x16 macroblock basis
Using an 8x8 DCT to clean up the 4 quadrants within a macroblock makes sense.
But performing a global wavelet transform on a motion compensated difference image
would mess you up at the boundaries of the macroblocks
since you would potentially have discontinuities there.
Of course it would be possible to devise a scheme
that had a different approach to motion compensation
like say a per pixel one that used optical flow.
Also, not all macroblocks are encoded using motion compensation
Even in P and B frames, some are encoded intra.
You would lose that small optimisation in a wavelet based scheme.
Similar techniques as the ones used to optimise DCTs
could be used for wavelets.
There has just not been a demand.
The standardisation effort tends to swamp out all alternatives
once the decision to choose the algorithm has been made.
There's a huge amount of momentum behind the standard
which makes alternatives very hard to pitch.
This is part of the reason why the same basic technique has been in prevalent use for 20 odd years and nothing else has come into play
