Where did the Black Stone held in Ka'bah come from?

Re: Where did the Black Stone held in Ka'bah come from?

Maybe the early Jews past through Mecca (Valley of Baca?) on their way to Jerusalem and it had some significance, like Muslims today consider Jerusalem sacred and make pilgrimage to Temple-Mount/Al-Aqsa on the way to Mecca?