You can do any to any just fine [open NAT ports, run your own distributed fallback network of TURN/STUN relays, add the adequate routing entries to your routing tables on both sides, exchange certificates, all of this for every extra N connections], you just probably don't want to do (and then fix if it doesn't work or stops working) that if N is too big.
However, outside some well-connected datacenters with multiple peering exchanges, I have no clue where in the world you can run everything in IPv6 with even a single nine in availability.
On my home I would say assuming single nine availability of IPv6 traffic is too much availability - it's very common that IPv6 is borked for several months in a row.
