I used to fret about it too but thank God not many ppl said anything to my face... the ones who did got the answer ...we are humans this is all in Allah's hands pls keep us in your prayers.... and I always had the look on my face which said pls do not probe me any further....
I just found out when ppl found out that we are expecting pretty much everyone thought that we could not have kids.... WTH.... so ppl will think what they want to think, without confirming with you.... so best answer is pls keep us in your duaas... and when Allah wills for us to have and raise kids so it will be....
Another woman who doesn't have any kids of her own ( I really feel for her poor soul, just a wonderful lady) had come to visit a family member who had just delivered and this woman looked at me and said : So automne you guys should do something to have a baby too now.... (I would've kept quiet and given her the diplomatic answer but there was my other relative who has always held it over my head that she has kids and we don't ) I just looked at her and sweetly answered: Aunty as husband and wife living together .... We do do stuff... it is upto Allah though to bless us with children and when He wants He gives.....
So you are gonna have diff situations just be patient...and pls don't rush into having kids just coz ppl are talking... pray that Allah gives you kids when you both are ready...coz ppl aren't gonna come and do your things for you...they are just gonna pressure you for the second one...
Sorry for the long post... wanted to share :)