I don't follow. If we're communicating with aliens that we somehow know are made of matter, we tell them that right is the direction a horizontal wire carrying positive current away from us gets pushed by a vertical magnetic field.
That just shifts the problem to specifying the direction of the magnetic field, and you can't use the concept of left vs. right in that definition without creating a circular dependency.
Vertically up it's just too a higher local gravitational potential, no? Sure, we flip right-left when we look at things upside down, but that's a nuanced usage.
How do you use closer and farther to determine whether a magnetic field is pointing up or down? It's not like there are little arrows on magnetic field lines you can just look at.