I think Sufis came to sub-continent before invaders. Who was the first invader of India and when he invaded Indian parts? Ghaznavis and Ghauris in 11th Century?
Islam always spreaded harmony? It never promoted intolerance. If we believe on this, its rejection of major facts and acceptance of generlisation of some incidents.
As I said above, these dynasties which are generalised as Muslim rulers had nothing to do with Islam. They were just rulers, who fought against the other rulers (who were Muslim too) just for the sake of rule.