The main technical underpinning (though note this isn't my area) seems to be basing indices on a particular improvement to B-trees, Cache-Oblivious Streaming B-Trees, which have some nice performance characteristics. I think what TokuDB calls "fractal trees" are just a catchier synonym.

Here's an academic paper from a few years ago from some of the people involved: http://supertech.csail.mit.edu/papers/sbtree.pdf

And a more recent talk focused on TokuDB: http://www.bnl.gov/csc/seminars/abstracts/Bender_Presentatio...