That's still dwarfed by a cache miss if you're unlucky.
I just found a bug (slow code is a bug) in the D backend where bad data layout led to 32 MILLION LLC misses (85% of the whole program) coming from one line!
Think about how much of a cacheline you are using per iteration folks.