I've never used helm, so I don't have any experience there. I can't say that I've felt like I'm missing anything by not using it though.

I'll give it a try on my system, maybe that will give some insight.

-- update -- I installed a bunch of helm stuff, it didn't slow anything down for me. try doing a clean pull from my repo and then adding your helm packages and see if that helps.

emacs starts up in about 1.2 seconds for me.