"I don't think there's really much of an analogy between programming languages and food."
Programming languages are cultural, not technical, artifacts. For reasons why one language is used by more programmers at a given point in time than another you need to turn to sociology. Lisp has some clearly superior technical ideas, which is why it has continued to be used and expanded for the last 50 years, and why all the other languages you mention (Python, Ruby, ML) drew ideas there. On the other hand, a language like Perl (and as Python and Ruby will eventually be) is briefly popular, but has no compelling technical features or metaprogramming facilities for people to continue to use and expand it when another shiny new language comes along.
Programming languages are cultural, not technical, artifacts.
Sociology is a factor but that's wildly overstating the case. The fact that people prefer to borrow from Lisp rather that adopt it is telling. Many programmers are aware enough of Lisp to take inspiration from it but choose not to use the language itself.
There's a subtext of arrogance in the claims FP and Lisp advocates make about the popularity of various programming languages. In my experience most of those people don't actually write much code. I gave up on the Common Lisp community years ago after realizing that it was full of people that would rather sit around discussing how things should be than actually building something.
A language like Perl ... has no compelling technical features or metaprogramming facilities for people to continue to use and expand it when another shiny new language comes along.
See the CPAN and especially modules such as PPI, Moose, and Devel::Declare.
And that still hasn't stopped the Perl 5 community from feelings of inferiority (see http://blogs.perl.org/users/ovid/2010/03/perl-5-is-dying-a-f... for example), and more importantly it hasn't stopped Perl 6 from being built. Which is a shame, because I would much rather have seen something like Perl on Rails, or in any case work going towards building on all the stuff already there in CPAN instead of duplicating all that work in PHP and then Python and then Ruby and then maybe even Perl 6 sometime this decade.
Why does Perl 5 need a Rails? Despite the existence of Rails, Perl 5 is still more widely used than Ruby. (Perl 5 and Ruby 1 came out around the same time.)
If you thought that was a joke, you're still missing the point.
I suffer no illusion that their sales numbers have any correlation to the popularity or efficacy of a technology. Their Ruby book sales crashed in 2007, for example.
Who is writing new projects in Perl 5?
I know of many, many new projects written in Perl 5. So far this year I've worked on three and have another later in the year.
Programming languages are cultural, not technical, artifacts. For reasons why one language is used by more programmers at a given point in time than another you need to turn to sociology. Lisp has some clearly superior technical ideas, which is why it has continued to be used and expanded for the last 50 years, and why all the other languages you mention (Python, Ruby, ML) drew ideas there. On the other hand, a language like Perl (and as Python and Ruby will eventually be) is briefly popular, but has no compelling technical features or metaprogramming facilities for people to continue to use and expand it when another shiny new language comes along.