Black magic is an ilm too. The Quran has mentioned its existence. That does not mean we are supposed to practice it. I think relying on God is enough.
If the prophets really wore gemstones, what makes you think they wore gemstones because the gems had the power to protect them? Did they not find God enough for them? Either you rely on God or you rely on a stone. If you rely on a stone, it means God isn’t enough for you. Either that or you believe that God will protect just because you wear a special stone.
If you rely on God, you don’t need stones, taweezein, etc.