Maybe the downside is that Bob will observe that others express displeasure at his behavior, having let down their expectations(maybe unreasonable) of him. Now Bob isn't callous by nature so this will weigh upon his conscience.