You're right. It was a consumption transfer away from Madoff's victims and to Madoff and his beneficiaries (with some blow-off to the rest of the economy when Madoff lost money in bad investments). Although there is some value lost whenever anyone invests in unproductive activities, because investment is an encouragement to those inefficient activities.
Theoretically, that would mean that none of the victims were defrauded of any money, or the ones that received money fraudulently could conceivably be forced to return it. But that's not my understanding of what has happened.
Actually, the earlier "victims" tend to make out pretty well, especially in the beginning. Later victims purely lost out. And Madoff certainly "skimmed" enough to live pretty well on.