Thats exactly what I do. Fortunately where we live (Salt Lake City) there are not too many desis so I dont have to worry about being spotted by some judgemental aunty. As long as my husband is Ok with it. My usual summer fare is shorts (usually knee or mid thigh length) with matching tank tops with either flip flops or sandals.
I own a few short dresses that I usually wear with pantyhose to formal events, most are either sleeveless or sphaghetti straps, have only one strapless dress that I wore just once, and will probably never wear again, was too uncomfortable in it, kept worrying that it would fall off.