Personally I consider time with friends and family productive time and most definitely time well spent, but surely you have noticed that even these moments can stretch out into unproductivity - maybe some members of the family or friend group want to go home and spend their time on their own things, so a little planning is useful even for family/friends/relaxation.
Productive usage of my time means I am not doing simple things the hard way, that I manage my attention, and that I spend time on things that I care about, not that I stop caring about my family.
Extending my example from above, American dining habits are: you stay in the restaurant for about as much time as what you need to eat/drink, then you go away (maybe gently nudged by staff to get out if you stop ordering something after 1 hour or so). Italian dining habits are: you go to dinner and you stay there until everyone has discussed about everything they wanted to discuss, no matter how long that takes.
I imagine for someone from the Italian culture your perspective would seem completely alien.
And regarding family members or friend that want to leave, surely everyone is mature enough to just get up and go when they have to. Why do you need to manage that for them?
Regarding my family/friends, I was talking about managing myself, my own time, not them - in this particular case I meant that I could predict when I can be home to do my own things, for example.
My perspective never disallowed me from staying at the table for hours until everyone was saturated with friendly banter. I spent many months in Italy, Croatia, Spain, Georgia, Poland and elsewhere doing just that, and I consider it time spent productively - relaxation and socialization is important to me.
Productive usage of my time means I am not doing simple things the hard way, that I manage my attention, and that I spend time on things that I care about, not that I stop caring about my family.