Unless you inline the CSS ;-)

This seems like it should work, but have you ever tried it? Or, can you point me to some results of a test to show that it indeed caches the image embedded in the CSS?

If it's inline, then it's cached with whatever it's inlined into.

The problem is that now it's going to be sent with every request. So it'll make the first page faster for the initial request, but slower in the long run.

I'm not sure how you figure that? The CSS file containing inline images would get cached, so nothing would be sent after the initial request.