I mean, it's like, 50 yards out of the way? You stop at whatever gas station you're driving past. I don't know anybody who makes a specific trip to go fuel their car.
The existence and popularity of sites like gasbuddy.com suggests that some number of people are willing to go out of their way to find the lowest price. I personally know people who will make a run to Costco for gas, even if they're not going to go in for groceries, because the price is good. There are at least a half dozen gas stations on the way to Costco.
Half the people driving out of their way to get gas are bad at math. The other half simply don't assign a dollar value to their time.
If your car gets 30 mpg, has a 16 gallon tank (that you refill at 1/4 tank, so you're buying 12 gallons), and you drive an extra 5 miles to pay $3.93/gallon instead of $4.00/gallon, how much did you really save?
I'll give you a hint: It's less than a nickel.
Meanwhile, you've probably driven at least 10 minutes that you didn't need to drive. 10 minutes to save a few pennies.
The math only gets worse as the gas prices go up and your fuel economy goes down. You need a greater delta to make the drive worth it.