Having a way to express Bayesian models is a requirement for probabilistic programming but that’s not the “point”.
Representing a Bayesian model in some language is not useful by itself. Being able to do efficient inference without requiring the programmer to implement an inference algorithm for each model they want to specify is the point of probabilistic programming.
I see that, so I should be thinking of the use of the phrase "probabilistic programming" to be more analogous to "Nonlinear programming" than to "Functional programming".
Or maybe more analogous to “differentiable programming” or “logic programming”: a style of programming that supports a broader range of operations than “running.”
Also, it’s not quite that you are using a DSL to encode Bayesian models traditionally represented in other ways; you are using a full-featured programming language to express a set of models larger than the set that is easy to express using existing formalisms. (That said, there are also some models that are easier to express using those other formalisms; you can think of them as being DSLs in a Turing-complete probabilistic programming language, which make it easier to express certain limited classes of programs.)
Representing a Bayesian model in some language is not useful by itself. Being able to do efficient inference without requiring the programmer to implement an inference algorithm for each model they want to specify is the point of probabilistic programming.