Because XP was feature competitive and there wasn't much market pressure to do so, unlike OS7,8 from that era.
This is quickly becoming a OS war, but I think its uncontroversial to say that the few years before OSX showed us an Apple that couldn't compete and was selling products that were substandard and overpriced. 95-99 was pretty rough.
I would say 1995 to 1997 were the dark days. Not because MacOS wasn’t working well (even though its aging architecture did mean it needed to be replaced someday.) Apple’s problem was one of focus. They had too many product lines, too many managers, and too many mediocre sales locations.
Steve Jobs’ return to the company changed all that. He killed most projects and fired a lot of people (after Gil Amelio had already done his fair share of firing.) Apple released great products again far before Mac OS X came out. The iMac was introduced in May 1998.
Microsoft didn’t release a new OS for 6 years (between XP and Vista) and even when it did, Vista flopped miserably. Yet, Microsoft is still around.