As someone who has studied post-Roman Britain as an amateur, I would guess they got on a boat and left Britain the same time the legions took off to play imperial whack-a-mole. That whole period of history is basically about migrations and population displacement, so it hardly seems unlikely that they could have effectively 'disappeared' over the next 3 centuries.