fazeelat that is punjabi and what the grandparents are saying is not at all indecent.
I am a punjabi and my grand parents always spoke with me in punjabi too. I don’t remember if my grand mom ever used urdu with me infact she didn’t know how to speak in urdu. I can tell you that me and all their other grand children have decent urdu & english accent and all of us feel proud to know one more language i.e. punjabi.
my cousins in states are more well versed in punjabi than urdu and they are just fine and to tell you 2 of them are even studying to be doctor.
don’t worry yours son will be just fine insha allah & remember your husband was raised listening to same language from the exact same people too !