I think leisure in that context refers to doing-what-you-enjoy not to not-doing-anything. Or as the old saying goes, if you do what you love you never have to work a day in your life.