Why do people worry about what "people" will say... they will talk regardless! If it's not the height it's the weight..if not the weight then any dimple, pimple, wrinkle!
As long as you like the person and the person likes u.. does anyone else matter?