It right now supports batch processing via local files as well as streaming through FIFO files, but the architecture is extremely open, and connectors for docker containers, mesos etc are planned.
What is awesome with SciPipe IMO is that half of it is not even a framework or engine, but rather a pattern of how to use the concurrency primitives in the Go language to build a dataflow network which also becomes an implicit workflow scheduler.
This means that there is not an aweful lot of code, and since it is mostly just Go language primitives it will fit in nicely with basically anything else. Great orthogonality you could say. That is, as long as you can accept to read and write on Go channels or write wrapper that does that for you. But who wouldn't?
It is a bit rough yet but we are just starting work on putting it in production later this month, so things should improve fast.
http://scipipe.org
It right now supports batch processing via local files as well as streaming through FIFO files, but the architecture is extremely open, and connectors for docker containers, mesos etc are planned.
What is awesome with SciPipe IMO is that half of it is not even a framework or engine, but rather a pattern of how to use the concurrency primitives in the Go language to build a dataflow network which also becomes an implicit workflow scheduler.
This means that there is not an aweful lot of code, and since it is mostly just Go language primitives it will fit in nicely with basically anything else. Great orthogonality you could say. That is, as long as you can accept to read and write on Go channels or write wrapper that does that for you. But who wouldn't?
It is a bit rough yet but we are just starting work on putting it in production later this month, so things should improve fast.