I understand what you mean. Scholars give an explanation for it that "stains are more prominent on white cloth". When because of good character of a person people develop good reputation for him/her, they don't tolerate even a minor mistake by him/ her as compared to the person who commits those sins in daily life as frequently as drinking and eating.
With time because of your good reputation people become more keen in observing you and don't tolerate even a minor act beyond their expectations, even if it is a misunderstanding, because they respect you. It is similar to a situation when a person is sitting in a glass house on the top of a mountain, from where everyone can observe them and point them out.
The only way to get rid of this criticism is to fall from the artificial mountain on the top of which people have placed you. Which means you have to ruin your reputation, because when you would be on the bottom of the mountain, a lot of your sins would not be clearly visible to common people because you are at the same level.
I know it is not good, but this is the way it works. And I am sure you don't want to ruin your character in order to save yourself from criticism. You have to live with it and ignore it even if people misunderstand your intentions. Because your intentions are not to please people but your ultimate LORD. Give explanations to only those who sincerely want to understand your intentions.
Very true!