Now combine this with an optimized SD implementation, like:

https://github.com/chengzeyi/stable-fast

Or AITemplate, and you are at 15FPS on a larger consumer GPU. 10 with a controlnet you can use for some motion consistency.