I think the exceptions to the rule are substances that aren’t eliminated by the body: small, “harmless”, doses can then accumulate with chronic exposure into a dangerous dose.
Not just that, but anything that is persistent in the environment and readily absorbable fits the bill.
It so happens that BPA and pthalates as part of the plastics made with these substances are persistent - they contain a high total and release the dosage extremely slowly into the environment.
A good example for that is also a slow toxic spill (of e.g. azo dyes) from a factory giving many bladder cancer. (See Pakistan.) The release is persistent due to persistent manufacture of clothing using the toxic substance.
Chronic exposures have a way of overwhelming or slipping by our internal detoxification - even though single dosage may be low and substance might not bioaccumulate, the damage does or probabilities get skewed towards certainty.
> a slow toxic spill (of e.g. azo dyes) from a factory giving many bladder cancer.
It should be noted in context that azo dyes are what the general public call "food colourings", at least the common, bright, inexpensive food colourings. And their accumulation in the body cause other types of damage as well, which is why there is no safe dose.
