Went back to do some more tests now, and funny enough I can actually get it to make decent stuff after realizing that it just completely sucks at below 512px (I was initially running it at 128x256 to speed up generation). I guess I should stop listening to advice from morons on reddit who said that lower res + upscaling works fine. Lol.

Not sure why there's even an option to go below 512.