this seems excessively reductive. by this reasoning we should expect to find that removing neurons from the "structure" has no effect on its capacity for encoding memory.

if one reduces "memory" to anything that exhibits sustained interplay between hysteresis and goal optimization then yes, neurons may lie below the level of explanation you wish to use, and could thus be substituted with some other signaling medium, but this is not the same as claiming they are not involved at all.