Dunno about weka but my last experience ( 5 months back) with Mahout was not good. There still are quite a few bugs and the fact that entire code base is in Java makes it extremely unpleasant for someone who wants to hack and modify the code to jump right in and start tweaking stuff. However, in its defense, it is open source is probably the only hadoopified ml library out there and has given me a ton of good ideas on how to write custom code.
Wow, we disagree. As much as I like to do my own development in dynamic languages like Clojure and JRuby, for me:
I would much rather have library and framework code that someone else has written, debugged, and supports to be written in Java: easy to browse in a good IDE, statically typed, lots of unit tests so you can hack away with some protection, etc.
Maybe my point wasn't clear enough:
1. I am comfortable with using someone else's library without having to reinvent the wheel but I want to know exactly what I am getting into without having to browse through tons of Java code. There are zillions of variants of algorithm X but I want to know exactly which implementation/variant Mahout uses without going through the source code. Unfortunately the docs (at least 4 months back) were pretty bad.
2. Their unit test coverage was not good enough which incidentally is how I found that there were bugs. The problem in trying to contribute back to the community by trying to rectify these bugs? When I read the source code, I get the feeling that each algorithm is owned to a great extent by one developer who brings in their own idiosyncrasies which means that you need to really study the code to make sure you don't accidentally add more bugs. The other disadvantage of this approach is that questions regarding potential bugs and puzzling issues can go unanswered or answered in an unsatisfactory manner (mainly because of the one developer writing most of the code issue).
Having said all this, I want to be charitable and chart these to growing pains. But if I were building something critical and big dataish, I would either use Python (dumbo) or Scala which are much more concise languages where it is easier to express math without introducing bugs.
