Speaking Urdu

This is where my concern comes. I love speaking Urdu and if I’m speaking to someone one-on-one I’ll speak in Urdu. But it’s the idea that do other people think I’m trying to hide something or exclude them when I speak to someone else in Urdu?

What if I’m doing a transaction with a guy and we’re speaking in Urdu and his manager wanted to do quality assurance on the level of service he’s providing or what if the manager’s concerned he’s giving me a deal/discount - I’m concerned with the lack of transparency.