Difficult question. I know that Matlab has some very strong Machine Learning libraries which make it extremely convenient to experiment. As much as I like GNU R and FOSS in this case I would say Matlab has the upper hand.

For one it has a huge userbase which posts code examples and solutions to various problems. Another being that Matlab has nice Machine Learning libraies in my opinion.

There are e.g. some interesting video lectures from stanford about this topic by Andrew Ng http://www.stanford.edu/class/cs229/ && http://www.youtube.com/view_play_list?p=A89DCFA6ADACE599 )

In the first lecture the question comes up and he has the same opinion.

I think eventually GNU R will be better but that will still take some time.

I find that programming in R is much nicer. I find that the language is more powerful and supports closures which is something I've always missed in Matlab. Extending R in C is also quite easy.

Its probably worth pointing out though that R is primarily for statistics. It walks all over Matlab in this domain (because most statistics researchers use R). However, for machine learning, signal analysis, system identification and working with ODEs I wouldn't be surprised if Matlab is easier and better supported.

R has a strong set of machine-learning algorithms, both official and contributed. There's a list here: http://cran.r-project.org/web/views/MachineLearning.html