Seems like a gigantic increase in the context window would be a great next step. GPT-4 is so damned good I tend to only become disappointed when it is forced to forget history.