I find these two options a must (they're in my .vimrc):

set hidden

set switchbuf=useopen

Beyond that I just use the basic commands to navigate buffers: :ls, :b, :bl, and ctrl-^

Ctrl-^ (switch to the "alternate" (previous) file) is the one a I use the most really. Especially since windows each have their own alternate file.

Generally, I find remembering buffer numbers in a long-running session no more difficult than remembering inventory letters in Angband ;-)