Perhaps the code itself couldn't be used by other vendors, but the ideas/protocols/interfaces could easily have been implemented by anybody who was so inclined. No trade secrets, no patents, and no dependencies on heaps of legacy cruft were necessary to be compatible. That's an important difference with "Embrace, extend, extinguish".
Ideas, protocols, and (except perhaps in the post-Oracle-Java world) interfaces can still be copied. A lot of open source software actually does exactly that with ideas, protocols, and interfaces from closed source software.
What makes open source software "open source" is the source code, which is subject to copyright law. Ideas, protocols, and interfaces are not subject to copyright; only their expressions are subject to copyright.
The Embrace, Extend, Extinguish approach makes things incompatible with the way others do them to cement hold on a large user base that prevents those users from changing their minds later. Sometimes that takes the form of forcing people to reimplement everything in a cleanroom setting, because they can never keep up that way; that's where the GPL excels within the open source world for EEE. Other times, it's by diverging from norms in ways that others cannot follow without breaking their own ecosystems, which is where the GNU project has excelled with its plethora of subtle incompatibilities.
