Interestingly, I recall the same problem as a young child, however, now I often need some form of "background noise" to focus.

When it's dead silent, I get distracted really easily.