I’d recommend using the ISCC-NBS system of color designation in cases like this. You’ll have to convert your colors to Munsell color space, and then use a lookup table to get the associated name. But it’s a pretty reasonable system.

Ideally you could get some kind of actual color measurements from clothes directly, instead of relying on (often very inaccurate) digital images, but maybe the’s too much to hope for.