We have an allreduce op in nGraph to support data parallel using OpenMPI, and are investigating the best approaches to integrate with frameworks. We also plan to add more collective communication ops to support model parallel in future.