Still, there are a lot of domains where transfer learning is no the most applicable setting - I'm thinking of highly noisy and non-stationary setting such as finance. In some of these domains, especially time series, lack of data is often not the issue, e.g. high frequency datasets.

Having models constantly re-train as the default setting is essentially what a rolling regression would do - having a rolling regression that doesn't catastrophically forget would be quite valuable.