Bhurban is somewhere close to murree or isb. i think, right? anyways, either tell ur husband how u feel and see wat happens then. cuz i can imagine ppl can get very annoying esp. when they r rude.
or like someone else suggested, go with them all this time and then go for another vacation some other time with ur husband and baby only.