3 or 4 stories is fine. Lots of the old parts of towns here in Italy are that high, and it's a nice compromise, because you can pack more people in, but buildings like that don't jut out from their surroundings when they're all about that high.