Most "side projects" have no users, other than maybe their creator, and therefore provide zero value to society. Furthermore, there's no intrinsic value in "contributing to society" — a term that is completely arbitrary, anyway. If the idea that you can improve society with your projects provides you joy, more power to you, but that's all it should boil down to: doing what you enjoy the most.
I agree that "contributing to society" has no intrinsic value. It is my own.
I feel like I waste my time when I play video games, but this is recent. I used to play a lot till age 24 or something. This feeling stops me from playing, but even if I do end up playing, I feel guilty afterwards that my time went to something that ultimately "does not matter". That said, playing something for fun is entertainment, it should matter, but I still get those feelings nevertheless. I hope this makes sense. I am not trying to argue that "fun" does not matter, I suppose I am just not so fun anymore. :P
It depends though, if I play a video game with my girlfriend it is alright.