The summary: if Coke and Pepsi are owned by different guys, Coke will take an action that makes them an additional $1 million, even if (especially if) it causes Pepsi to lose $1 billion. Most commonly, cut prices.
If the two companies are owned by the same guy, they have the incentive not to compete with each other since their owner cares about the sum of their profits. This is why one wouldn’t be allowed to acquire the other. But the effect is the same when a single index fund owns a large chunk of both companies. The companies are encouraged to compete not too fiercely with each other.
Even without index funds the drive towards large portfolios and diversification would do the same thing. I'd go so far as to say that unified ownership of competing firms by large investors is just unhealthy in general. Not really sure what can be done about it though since many companies are rather multi-industry.
Well the idea of an index, whether done through a fund or not, is to buy some of each company. A value investing approach would be to find the better stock and invest in it, so Coke stockholders would rarely own Pepsi stock and vice versa.
Still, even without index funds, the first thing most people are told is that diversification is key and that sector allocation matters more than the specific companies you hold. Plus people want to mitigate risk, so they buy a little of each thing that matches whatever profile, and soon a sector essentially has unified ownership interests rather than competitive (and antagonistic) ownership interests.
If Coke could take an action that would make them $1 million but cost Pepsi $1 billion, why wouldn't the two companies come to an agreement for Pepsi to pay Coke a one time sum of X where X is between $1m and $1b for Coke to not do it? A single entity owning both corporations should not be necessary for the efficient outcome to occur in this hypothetical.
These are not negotiated activities operating in an environment of collusion. They are the result of separate corporations operating in an environment assumed to be competitive. Individual corporations will take strategic actions that may harm their competitors disproportionately to the benefit they receive. It is not zero-sum, and would be economically value-destructive for shareholders of both (while possibly beneficial for consumers).
If the two companies are owned by the same guy, they have the incentive not to compete with each other since their owner cares about the sum of their profits. This is why one wouldn’t be allowed to acquire the other. But the effect is the same when a single index fund owns a large chunk of both companies. The companies are encouraged to compete not too fiercely with each other.