These aren't new problems. Companies have been solving and dealing with this sort of problem for _years_. It's how a lot of Amazon retailers, and ebay retailers, and NewEgg retailers, and Rakuten.com (Buy.com) retailers, and Google Shopping work.

There's a billion different extensions just for Magento for "Product Feeds."

If you try to make it even more realtime, may god have mercy on your bandwidth.