Would it be possible to create a container for every site I start at in a fresh tab.
E.g. I open a tab, go to example.com and that stores all cookies and cached images and so on from that point on in a container that get's opened every time I start at example.com. So you have one container for every start point.
Possible in principle? Sure. Possible with the current state of the implementation? Definitely no. It's currently not even possible to create your own containers, and instead there's 4 pre-defined ones.
But the Tor Browser has something which comes kind of close to / goes beyond what you what you want.
Basically, what they do, is that they have a separate container per domain. So, if you are on www.example.com, and the Facebook-Like-Button on that page sets its Tracking Cookie, then this Tracking Cookie is tied to both facebook.com and example.com. So, if you then browse to www.yetanotherexample.org, then the Facebook-Like-button on www.yetanotherexample.org won't be able to see that Tracking Cookie, because the Cookie is only in the example.com-container.
I figure that this doesn't solve what you want solved, but it shows that it's definitely possible from a technical viewpoint, and then having a small script which instead creates a container in the way that you described, that should be trivial and easily doable from an extension.
E.g. I open a tab, go to example.com and that stores all cookies and cached images and so on from that point on in a container that get's opened every time I start at example.com. So you have one container for every start point.