I'm really not a medical expert at all. But, just thinking about it one could at least speculate or argue: A smaller payload means the virus takes a longer time to "become very active" in the body and delaying this "ramp up" period may give your body's immune response a longer lead time from when it first "sees" the virus to when it becomes dangerous to you.
Well with an exponential curve, even if the amount of the initial dose changes 10 fold, from 20,000 to 200,000 virons you will get to hundreds of millions in pretty much the same time. I believe adaptive immunity takes about a week or two to kick in, and in that time the virus is going to keep multiplying like crazy, regardless of the dose.
Also, suppose you were living with a partner and both of you got infected. Both of you will continually shed and "re-infect" the other, so would we expect that the infection will necessarily be severe?