And, in rare cases board members can be held responsible for the actions of the board. In that sense you have a lot more to lose as a board member than as a shareholder.
So if you think that the board might be held liable by shareholders for something in the future it is usually a prudent step to create as much distance between you and the board as a shareholder as you can so you don't end up on both sides of a lawsuit.
Is that true? Many people are on multiple corporate boards, and some boards meet only a few times a year. I've only been on charity boards, but it wasn't a particularly demanding commitment.
I guess the reputation cost is a good point, but Kalanick is always going to be associated with Uber, whether he's officially on the board or not.