This is a misstatement.
Hazrat Imam Ali (as) did not give allegience to the caliph. With him remained from the very beginning the most prominent and close companions, like Ammar ibn Yasir, Miqdad bin Aswad, Salman Farsi, Abu Ayub Ansari, Jabir bin Abdullah, Saad b Ubaydah, Banu Hashim the tribe of our Prophet (saww) and more who were all supporters of Hazrat Imam Ali (as).
actually sa'd b ubaydah radiallahanhu wanted to be caliph himself thats why he and his supporters gathered at the saqifa even before the muhajireen got there ...
there were other supporters of sayyidna Ali, but there arguments for their choice are very different from what 12ers base their claim of succession