Well, that was mainly due to the fact that Ansar had anticipated what the Quraysh were upto and ofcourse they did not fancy losing the control of their own town to the people who they did not trust.
And If you read further then you will find that Ansar at the same gathering had said that they will not give oath of allegiance to anyone except Imam Ali (as).