The benchmarks are based on old Julia--something I've wondered about--but they still are consistent with my experience (and with the benchmarks the OP posted).
I really like Julia, so I don't want to come across as a hostile critic, more of a friendly one. I'm someone who has, and continues to try, Julia for lots of things. But repeatedly I've bumped into this phenomenon where Julia is maybe 3-5x slower than what you'd see in C, maybe more, and am faced with what that means in practice. It's not just Turing, and it's not just a single version of Julia. It's a heck of a lot better than Python or R, but when I realize it's still significantly faster to do something in C, and you're going to wrap something around C anyway, maybe Python or R would be better just because of the resource base associated with them.
Maybe there's some fundamental issue I haven't grappled with, like Julia will never actually be as fast as C, and I just need to be aware of what that means, practically speaking... I don't know. Maybe it will eventually hit that performance range, and it's just a matter of optimizing algorithm choice and whatnot. But I get the sense that the way I was hoping to use Julia isn't actually doable, at least at the moment.
I do think many people are coming to Julia because of some sense they can get the performance of C with the expressivity of Python, and at the moment I think that's an unrealistic expectation for a lot of things. I think the parent post in this subthread exemplifies that quite well: for many, many reasons, it shouldn't be surprising that Stan outperforms Turing. If this was any other language, it would be a matter of course. But because of the expectations set up for Julia, you see discussions like this.
If you can turn one of those trials into a small example and post it on the forums, it'd be well worthwhile. We've often helped folks squeeze the last few bits of performance out of their code – and really, anything past 2x of C is considered a bug.
I really like Julia, so I don't want to come across as a hostile critic, more of a friendly one. I'm someone who has, and continues to try, Julia for lots of things. But repeatedly I've bumped into this phenomenon where Julia is maybe 3-5x slower than what you'd see in C, maybe more, and am faced with what that means in practice. It's not just Turing, and it's not just a single version of Julia. It's a heck of a lot better than Python or R, but when I realize it's still significantly faster to do something in C, and you're going to wrap something around C anyway, maybe Python or R would be better just because of the resource base associated with them.
Maybe there's some fundamental issue I haven't grappled with, like Julia will never actually be as fast as C, and I just need to be aware of what that means, practically speaking... I don't know. Maybe it will eventually hit that performance range, and it's just a matter of optimizing algorithm choice and whatnot. But I get the sense that the way I was hoping to use Julia isn't actually doable, at least at the moment.
I do think many people are coming to Julia because of some sense they can get the performance of C with the expressivity of Python, and at the moment I think that's an unrealistic expectation for a lot of things. I think the parent post in this subthread exemplifies that quite well: for many, many reasons, it shouldn't be surprising that Stan outperforms Turing. If this was any other language, it would be a matter of course. But because of the expectations set up for Julia, you see discussions like this.