It's about compromise. A smartphone with a one inch screen would be an obviously stupid idea, as would a smartphone with a ten inch screen. Seen that way "bigger is better" or "smaller is better" doesn't even make sense.
My claim is: 3.5 inch screens are a good compromise, allowing for comfortable browsing, reading and writing. The additional screen space would never be worth the additional bulk (also: uncomfortable one hand use) for me.
I think something between 3.5 and 4 would be ideal. Current 3.5 size is small for reading books(typical pdf technical book, page cropped to visible content). Even adding 0.25 inch would be a great help without sacrificing portability.
Another thing to consider is in general the screen bezels of phones is getting thinner too as the screen glass size grows, a 3.5" (88.9mm) iPhone has probably a 1/4" (6.3mm) bezel surrounding the screen.
Someone with an iPhone would be used to the "hand feel" of a 4" (101mm) iPhone if the screen bezel were thinner.
My claim is: 3.5 inch screens are a good compromise, allowing for comfortable browsing, reading and writing. The additional screen space would never be worth the additional bulk (also: uncomfortable one hand use) for me.
That's all about personal preference.