The main technical underpinning (though note this isn't my area) seems to be basing indices on a particular improvement to B-trees, Cache-Oblivious Streaming B-Trees, which have some nice performance characteristics. I think what TokuDB calls "fractal trees" are just a catchier synonym.

Here's an academic paper from a few years ago from some of the people involved: http://supertech.csail.mit.edu/papers/sbtree.pdf

And a more recent talk focused on TokuDB: http://www.bnl.gov/csc/seminars/abstracts/Bender_Presentatio...

We have some more descriptions of Fractal Tree Indexes at http://tokutek.com/what-is-a-fractal-tree, and I'm always happy to answer questions about it.

For starters, it totally changes the database performance paradigm for managing tables with many indexes.