Usually it's when expectations change. In my case, I took on new responsibilities and pushed to be compensated accordingly. I was then held to higher expectations which I was unable to meet. In my case this was mostly doomed fail from the beginning because my responsibilities were in conflict with the company's needs. Small startup, poor management structure, pre-revenue, etc. all contributed.

Your metric of "meet performance goals, get along with manager and skip, business doing well" is a solid way to track things, especially earlier in your career.