Yes. Though when you assume something like an efficient market, variance becomes more meaningful.

Under some suitably broad efficient market assumptions, the expected return for all stocks is basically the same (after adjusting for risk). So variance automatically means going up and down compared to that expected return.