Yeah, Apple was right. Duh.
As pointed out here and here and other places, 3.5" screen is just the right size for a normal adult to comfortably hold and use the iPhone in one hand. Thinking about it as I use it, I'm able to touch the farthest points just at the point before it would be uncomfortable. Even another 1/4 inch away for the farthest corner gets pretty uncomfortable.
Before the first iPhone came out, there were zero points of reference, and they could have made the screen any size or shape they wanted. I'm sure there were lots of prototypes. At another company, you'd expect there were committee meetings and focus groups, but I'd bet at Apple, Steve Jobs just handled them all and picked the one because that's what fit his hands. And the 3.5" size was THE size that was as big as possible while still being usable one handed.
And that's the one and only size the rest of the world ever saw.