Good point. Only about 50% of Britons voted in the Brexit referendum, for a turnout of 72% [0].
So the inequality is twice what I made it out to be there. I don't think it changes my comparisons all that much: we don't usually consider infants when measuring inequality in wealth or educational attainments either, or we use their caregivers as a proxy.
People who didn't vote, People who are too young to vote, People who cannot vote. Etc
Voters as a group is meant to reflect that proportion of the population that is involved with the political process at the basic level.