FWIW I have always been under the impression that "first of all that's rude" is mostly used as a funny meme, a recurring harmless little joke (to put emphasis on what comes next in the speech).