Maybe this is an American thing. In Australia and Europe I don't think I've ever heard someone say something like this. I've heard people complain about being homesick or tired if it's a particularly long or active holiday but never heard someone wanting to get back to work.
I'm Australian and I am often very keen to get back to my desk to work on ideas towards the end of a holiday. My side projects somewhat interrelated with my work, so maybe that's one reason.
