FYI Tortoise, the thing you are trying to build, is abandonware. The creator decided to stop working on it due to “ethics” (i.e only Big Tech should have access to AI) when the community reverse engineered a way to finetune it using weights accidentally left on hugging face. There’s a nice fork out there called mrq/ai-voice-cloning.