The three servers are federated with each other and the one of them is federated to the broader matrix.org network.
I've found that loading the larger rooms from the matrix.org network can cause a bit of lag for the few dozen or so users on my servers but once the rooms have synced up that issue goes away.
I don't administer the servers personally so I don't know exactly what the issue is but I have a feeling it's due to the limited ram that is provisioned to the VMs running the instances of synapse as there are many other VMs running on that same laptop.
If the instances of Synapse were the only things running on the laptop and they had 12gb of ram each (1/3 the total 32gb limit of the laptop) I would imagine that this issue would go away.
Is seems that this just helps the argument of how inefficient Synapse is. 12GB ram is quite beefy server. the issue is that when you want to federate your server needs to be able to handle similar loads as the server you are federating with....
So not like email where you can have tiny server and still be part if the network.
I have an old laptop running a bunch of different services, three of which are matrix servers. The matrix servers generally run fine with the modest amount of ram provisioned to them (2-4GB IIRC) with the exception of hiccups that occur when a user joins large federated rooms (we're talking several thousand users) and these hiccups only last for a few seconds and until the room has finished syncing to my server. The hiccups manifest as "Unable to connect to server" messages on clients which only delays messages from being transmitted to the server by a few seconds.
From my understanding once a single user has joined these rooms and triggered the server side sync these hiccups will not happen again once another user on my server joins the same rooms as the data is already synced on the server. Furthermore there are relatively few rooms that have the number of users necessary (1000s) to trigger the issue one time join issue.
To remedy this issue and for general network reorganization purposes I intend to migrate non-synapse VMs off of the laptop and onto other hardware and then I would simply delegate the old laptop for the matrix servers and scale the three VMs to give them 1/3 of the available ram whether they need it or not. I mispoke last night when I said 12gb by the way, as the laptop only holds 32gb ram so the available ram could be (32 - (ram for hypervisor) / 3)
To bring it all back to the thing I care most about which is the over all cost, if the laptop cost $300USD and I run 3 servers on it my hardware cost per server is $100 which I consider to be trivial. This is 20 cups of fancy coffee and more importantly many hobbyists already have a spare laptop sitting around or can find one for quite an affordable price.
I've found that loading the larger rooms from the matrix.org network can cause a bit of lag for the few dozen or so users on my servers but once the rooms have synced up that issue goes away.
I don't administer the servers personally so I don't know exactly what the issue is but I have a feeling it's due to the limited ram that is provisioned to the VMs running the instances of synapse as there are many other VMs running on that same laptop.
If the instances of Synapse were the only things running on the laptop and they had 12gb of ram each (1/3 the total 32gb limit of the laptop) I would imagine that this issue would go away.