You're overestimating the level of technological sophistication on the retailer side. Many of them not only don't have any "retailer APIs", but they have no internal view of their inventory - so if you want to find out whether a given store has a given SKU in given size, the only option is to call the store and have the guy check in the back.
A middleman service with a standard API plus the push from consumers / front-ends might encourage them to build that out. In any case, I wasn't thinking individual meatspace retail stores; I was definitely thinking more "whatever is sold on the web site now" (e.g. Macy's, Asos', Urban Outfitter's online stores - which I'm making the hopefully-not-wild assumption have their own separate inventory.)
Tackling per-physical-store retail inventory with the challenges you mention is definitely a whole other thing.
