OK found the following article which explains the reasons for their expulsion. The few non muslims were allowed to stay as long as they stuck to their agreement, but were expelled when they broke it.
Source
Expulsion Of Jews And Christians From Arabia
At the time of his death the Holy Prophet had expressed the view that in Arabia there should be only one religion, namely Islam. During the lifetime of the Holy Prophet, all the tribes in Arabia had accepted Islam. There were only a few pockets of non-Muslims, the Jews in Khyber, and the Christians in Najran.
During the caliphate of Abu Bakr, many tribes who had accepted Islam apostatized. As a result of the apostasy wars, all the apostate tribes were defeated and they once again accepted Islam. During the brief period of his office, Abu Bakr allowed the status quo to continue with regard to the Jews and the Christians.
At the time of the conquest of Khyber, a treaty was executed with the Jews hereunder they were allowed to cultivate the lands on the payment of one half of the produce to the Muslim state at Madina. The treaty also provided that the Jews could be turned out of Khyber, whenever the Muslim state deemed it necessary.
When Umar became the Caliph he deputed his son Abdullah to Khyber to collect the revenue. As Abdullah lay sleeping on the roof of a house in Khyber at night, his bed was overturned by the Jews causing him an injury in the arm. Umar investigated the matter and found that the Jews were bent on mischief. Umar accordingly passed orders expelling the Jews from Khyber. They migrated to Syria. They were allowed to carry their movable belongings with them. Their immovable property in Khyber was distributed among the Muslims.
The Christians of Najran near Yemen had a pact with the Holy Prophet "hereunder they were allowed to live in peace unless they indulged in any hostile activities against Islam. It was also stipulated that they would not indulge in usury. When Umar became the Caliph it was brought to his notice that the Christians of Najran had violated the peace pact in as much as they were indulging in usury, and were also guilty of activities hostile to Islam.
Umar summoned the representatives of the Christians of Najran, and apprised them of the charge of violating the terms of the treaty. In a vainglorious mood the deputationists said “If that was that, they might be expelled.” Umar accordingly passed orders for their expulsion. Arrangements were made for their settlement in Iraq. They were allowed to carry their entire movable property with them. Their immovable property was acquired by the state on payment.
Umar instructed his officers in Iraq that all possible assistance should be provided for the settlement of the refugees from Najran in Iraq. The Christians were exempted from the payment of Jizya for the first two years.
With the expulsion of the Jews and the Christians from Arabia, the country became an exclusively Muslim land. Umar has thus the distinction of being the first ruler under whom Arabia became the exclusive preserve for Islam.
ordered the expulsion of jews and christians from Arabia, however in the conquered territories jews and christians were allowed to practice their faith freely.