I agree, but I got pretty far towards this when I got fzf's ctrl+r replacement to search history quite nicely.

Before I used to also have a per folder history file, so I could pull up the last thing I did there. That also helped a lot, especially when switching between different personal projects and forgetting how it all works.