All of what you describe is probably a big part of why we don’t remember. Not only has the though encoding/decoding process not been developed sufficiently to remember things in the way we do when older but even if we did those memories would probably be horrific for the most part.
i think encoding and decoding has a lot to do with it. we can watch a Disney movie and I'll ask my 4yo what it was about and she can name the main character but without prompting can't tell you where they went or why. Those phony sounding assignments of "what i did over summer vacation" don't seem so useless now. Right now we're getting lots of imaginative story fragments but missing a coherent narrative linking one sentence to the next. It's interesting watching them play with language and learn how stories work in real time