Are you talking about parties then its not fine but if some one is going out for golf or some kind of sports activites and he has to pick something from you then you shouldn’t mind that. 
No gora person ever showed at my home in shorts not even the construction workers, painters etc
Ofcourse they cannot its all about dress code ke kahaan hain and kis mahol mein hain still I don’t get your point aap aur clear karo.ab swimming pool parties mein to koii pant ke saath swimming kernay se raha 