If you've ever actually burned a fleece accidentally you will find they just shrivel up a bit, they don't catch on fire very easily unless you throw them in one.

When they "shrivel up" the fabric is actually melting. Melting into your skin is what turns a minor burn to turn into a life-threatening 3rd degree burn. "catch on fire" is NOT the danger.

As you say it does not catch on fire, there is no danger in wearing a fleece while sitting next to a camp fire; there is a danger in wearing a fleece while sitting in a camp fire.

I've worn a fleece that got burned while I was wearing it, one time while tending to an open fire, another time with a soldering iron, in both cases it clearly did not want to propagate, I just ended up with a small patch of shrivelled up fleece/hole that didn't stick to my skin, in fact I'm pretty sure the fleece saved me from a burn in the case of the soldering iron... I'm sure that if you were fully engulfed in flames for long enough it would start to stick to your skin (which is not exactly an exclusive property of that fabric), but that's not what you were suggesting.