so here u are saying u are not sure what the reason is i.e. whether his social ettiquette and manners do not match up to your expectations, or that he finds it dishonorable to talk to you.
However, you extrapolated it to the entire muslim men population in your initial post, so its evident that you think its the latter.