I usually just use local spark for some large file that pandas cannot handle well. It's not perfect but it works for me.