What if also have an energy term for difference in interframe x-position along the seam?

Or maybe doing histogram equalisation for pairs of adjacent frames before the seam cutting might help.