It depends on the situation but I pretty much agree. For example, sometimes coworkers will come up to me to discuss or ask something that'll take less than a minute. These minutes can add up to suck all the productivity out of your day, especially with the context switching.

On the other hand, something that could take a minute for me to answer could take someone an hour to figure out, thus thwarting productivity on the other end.