The size of the dose is not a great measure of safety. Botulinum toxin has an LD50 in the nanograms, and several other toxins in the same range of LSD, that being micrograms.

Now no strong evidence of toxicity has been found from LSD, but that's not due to it being taken in small doses.