Not to dismiss your rebuttal (I don’t believe the technology for voyages cross the south Pacific existed in the Americas [south of Greenland] prior to European contact either), but it is also possible that some unlucky fishermen got caught in a storm off the coast, and was unable to return to land. They miraculously managed to survive while being carried with the currents to e.g. Rapa Nui where they were saved by the local population, nurtured back to health, and lived there, had children and then died.
What evidence would we expect to find? I would imagine any wooden boats from that long ago would have long ago disintegrated by water, tropical jungles, tropical fauna, etc. long ago. Lack of proof isn't proof of non-existence.
Oral history, anchor stones, parts or fragments of vessels (carvings from the prow, steering oars, and so on).
In Polynesian culture, an ocean-going vessel was a thing of great mana, having its own name and celebrated in many stories. Relics were revered and carefully preserved.
I don't think Polynesians are unique in this regard.
Isn't it easier to assume that both groups reached Easter Island independently and reproduced on the island with each other?