This is a really hard subject. I can remember growing up as the only black kid in an all white class of kids in elementary school, and all of my friends made it into the "advanced" cohort. I knew I was just as bright as any of them, and that given a chance I would thrive there too. But for whatever reason I didn't make the cut, and this removed me from my peer group for the rest of my school years. They grew apart and I ended up mostly friendless. I don't think race really had a lot to do with it. But if you're going to separate kids, at least have a whole other school to send them to, rather than creating a distinct hierarchy among the classes. I think that really messes with kids' self esteem.
In my advanced program, they would ship us to a nearby elementary school, and the advanced students there would be sent to our school (same district, same grade level, just 10 minutes down the road), I can only assume it's for the same reason you mentioned...