As a working physicist I would agree with you for the most part. Science fiction doesn't help because it doesn't contain ideas, which work beyond a surface plausibility level. However you can still get inspired by them in indirect ways. I find Paul Feyerabend's work interesting in that regard. In "Against Method" he argues that most breakthrough scientific discoveries can't be cleanly attributed to the scientific method.