I've variously heard it was a 5% commission for the groom, jockey, or auctioneer. If a horse won a 100 guinea prize, the owner conventionally got 100 pounds and someone else got 100 shillings in commission.

I don't know about this aspect, but horse sales typically involved this type of commission. The buyer would pay in guineas (or £1.05 times the nominal purchase price) and the seller would receive that number of pounds, with the extra going to the auctioneer.

I'm not sure if this system persists now, but it definitely lasted long past decimalization.

And oddly, in auctions for farm animals. If you go to buy a bull at an auction mart, you bid guineas, not pounds.