These fitted ones are flush with my ear (actually slightly further in even), so not had that issue. Sometimes putting the helmet on knocked the plug and meant the seal wasn't quite right, but easily solved by putting it back in!

Screen and helmet definitely make a big difference to buffeting though; funnily enough not always a bigger screen, just a different shape I found.