May be my AI tasks are to simplistic, but I never had problem training AI on single machine and as many pointed out there always cloud services, otherwise I find it an impressive work, it takes some courage to say well there so many huge companies with there's frameworks, but we can outperform them all in specific problems.

Couple things that stood out from github page:

Currently we only support homogenous multi-agent envs (same observation/action space for all agents, same episode duration).

For simplicity we actually treat all environments as multi-agent environments with 1 agent.

My speculation is that this why they gained such dramatic performance improvement. (but I might be very wrong)