If you think our behaviors and our perceptions about those behaviors are dictated by circumstance and environment we got then how come several people with similar features exhibit similar pattern of attitude toward others? Sadness reflection, worried, angry, or arrogant is simple thing, anyone can get that and that does not modify our facial features.
I am not saying that it is an absolute fact that people with similar facial features always show similar attitude/emotions but there are things they share in common and then there are exceptions too. I would try to explain those exceptions.
Yes, there are exceptions and we do not know all of them. For example, analogy of one feature can negate the analogy of other feature. Might be, some behaviors could be identified by adding up analogies of two or more features.