That's a problem that affects all theories about the metaphysics of consciousness. All of them share the burden of explaining how a world where "consciousness" exists would be different from a world without it. The ones which claim a distinct consciousness does exist then also have to explain how this consciousness either originates from physical parts with limited or non-existent consciousness(emergent theories like panprotopsychism and emergent materialism), exists outside physical matter (dualism), or exists fully in the smaller parts, somehow getting combined into one (traditional panpsychism).
That's part of why it can be attractive to leave aside concerns about metaphysical consciousness altogether.
If brains are information processing systems, and that consciousness emerges from that activity, I don’t have to worry about the parts having consciousness. They don’t, they just contain and process information.
Panpsychists are committing a hierarchy inversion error. Horses are a kind of animal, but that doesn’t mean all animals are a kind of horse. Just because consciousness is a result of information processing, that doesn’t mean all information processing systems are conscious. Heck, as we are discussing here, I’m not always conscious.
If we define it to be just a human interpretation of an observed physical process, and we believe subjective experience to be an illusion, then the hard problem is not a problem at all. That's comparable to calling an animal a horse: we define "horse" to be a category of animal, but we don't expect it to have any "special" properties for being a horse, and we can relate to lower-level processes all the properties that lead us to define it as a horse. Such properties are empirically measurable and don't behave in any way that would be expected from the laws of physics.
But if we assume subjective experience to be real and to be more than just a description of the physical system, then we do have a problem: we have the burden of explaining where it comes from (which is why theories like emergentism, panpsychism, etc. were created). We can't test such theories empirically, as the only subjective experience we can perceive is that of a fully conscious (in the medical sense) human brain, and we cannot share such experience. That's why the metaphysics of consciousness is not considered science.
> Just because consciousness is a result of information processing, that doesn’t mean all information processing systems are conscious.
Indeed. But if consciousness is anything more than a label to describe a specific sort of system, and subjective experience is real, then we have to show such special "consciousness" (with subjective experience) exists in the first place, and how it can emerge from parts without any form of consciousness. As we're unable to do that in a way that can be empirically tested, such theories are not scientific. Panpsychism is of course no exception to this, as we cannot test whether lower levels of information processing are or are not conscious either.
That's part of why it can be attractive to leave aside concerns about metaphysical consciousness altogether.