Guess you need to meet more people 
I know plenty of everyday folks who rock the pattern.
check this link out from askmen, they feature banana republic plaid trousers
Banana Republic Plaid Wool Trousers - AskMen.com
and then banana republic, you see three trousers, glenn plaid, window pane etc.
as I said, colour, cut and what you pair it with will decide whether it works or not.
Apparel, Handbags, Shoes and Accessories for Women and Men | Banana Republic