This is all personal opinion, of course. I think the key to what makes Riak great is that it is fully distributed. Every node is a peer, and this eliminated single points of failure. But this also makes things a challenge for organizing work. Riak was previously a fairly monolithic product[1] with a set of features, including being a KV database and doing Map-Reduce processing. At some point Basho, wisely, decided that making the product more modular would allow them to be more agile in their development.

So, they split the KV database from the ring code, creating Riak_Core and Riak_KV. Riak_Core allows you to crate a ring of virtual nodes on a cluster of physical nodes, and spread work around it. (essentially the dynamo concept.) Thus, Riak KV then became an application running on the virtual nodes of Riak Core, providing a key-value database. At this point (e.g.: post Riak Core split, but pre-Riak 1.0) Raik_KV also managed the Map-Reduce functionality.

With Riak Core, you can create an application that does whatever kind of work you want, and spread it around a dynamo style ring. The ring is just a way of partitioning up work using a hash function so that it can be evenly distributed across the virtual nodes (Which are cleverly distributed across the physical nodes in the cluster.)

Riak Pipe is an abstraction on top of Riak Core that makes lets you build a pipeline of processing. Each stage in the pipeline is called a fitting. Each fitting has a function (that does the work) and a function to decide which vnode to do the work on. When the pipeline has data going thru it, the vnodes that get work create queues and worker processes to do the work. A key feature of this is that if a queue gets full, earlier fittings in the pipeline are stopped from adding to it, such that their queues will eventually get full (say if there's a very slow process near the end of the pipe) producing a "back pressure" to prevent work from overwhelming the cluster (or a particular vnode).

So, for Riak 1.0, they re-worked their Map Reduce implementation to run on Riak Pipe. This will allow for more flexible map-reduce jobs in the future (maybe even now). As an example of how the map-reduce implementation works, a map phase might be described as a fitting that uses a word-count function (to do the work) and uses the hash of the piece of data from Riak KV to determine which vnode on which to run. so, as you fill the pope with documents to have their words counted, the tasks get spread to fittings across the cluster, and then each fitting sends its results to the appropriate vnode for the next stage in the pipe (which might be reduce) ... and here's the key point... without it having to talk to the node that started the job. Previously, the node that started the map-reduce job (I believe) had to coordinate it across the cluster)... now it self coordinates.

The great thing about Riak Pipe, though, is that it is (as I see it) essentially a realtime processing engine. Say you had a job where you were monitoring the twitter firehouse for mentions of your company. The task is relatively straightforward, but you wouldn't want it to all be running on a single node, right? So, you'd have the fittings work function be the code that scans for your company name in the tweet and flags it, and the function that determines which vnode to run on could be a random hash (so it's evenly distributed across all vnodes.) When the firehouse overwhelms your cluster, you don't find yourself swapping because back pressure will stop new tweets from going into the pipe, and if you need to add capacity you just add a new machine to the cluster.

I'm still wrapping my head around some parts of Riak Pipe. I think that it will turn out to be a really killer feature.

[1] Seems silly to call any cluster of a bunch of erlang processes "monolithic", but a better word is escaping me at the moment.

Great explanation, thanks!

Interestingly, the first time I saw Riak I was interested in its potential for becoming a stream processing engine. Now, they are following this path. This is a very timely move as there's an increasing interest in this area and many emergent technologies like Yahoo's S4 and Twitter's Storm.