Of course it doesn't speed up your own time, since you have perfect information about your own hashpower. But it does tell you information about the total hashpower that's online, statistically.

I'll give an extreme example to make this clearer. Suppose 10X hashpower just came online an hour ago. It's quite likely that ~60 blocks have been found in the last hour, assuming the difficulty adjustment hasn't happened since. Seeing this, one could deduce that hashpower went up by ~10 and that the expected time till next block is roughly 1 minute instead of 10.

Now, in most cases hashpower doesn't change that drastically but it remains true that recent block times give you more than 0 information about hashpower and therefore about the expectation for future block times.