That's a fair comment RE: outsourcing, but at my company I'd bias towards bringing some distributed computing knowledge in-house rather than bringing ops expertise plus maintenance burden in-house; sounds like you're going to have to add new expertise to your team either way.
Worth investigating if you can bolt on a distributed datastore like etcd or ZooKeeper to store the cluster membership and data locations; this might not be as complex as it sounds at first. etcd gives you some very powerful primitives to work with.
(For example, etcd has the concept of expiring keys, so you can keep an up-to-date list of live nodes in your network. And you can use those same primitives to keep a strongly consistent prioritized list of repos and their backed up locations. The reconciliation component might just have to listen for node keys expiring and create and register new data copies in response.)
Think about it this way: your EE customers probably have the easier bare metal knowledge, but would be willing to pay for you to solve the distributed system problems for them :)
Worth investigating if you can bolt on a distributed datastore like etcd or ZooKeeper to store the cluster membership and data locations; this might not be as complex as it sounds at first. etcd gives you some very powerful primitives to work with.
(For example, etcd has the concept of expiring keys, so you can keep an up-to-date list of live nodes in your network. And you can use those same primitives to keep a strongly consistent prioritized list of repos and their backed up locations. The reconciliation component might just have to listen for node keys expiring and create and register new data copies in response.)