since you are under an oath, dont initiate a break up .. but behave in such a way that he is compelled to break up himself 
and yes, THEN it will be b/w him and his God (like he said too) !!
in short, he is not worth a relation. even if you two get married eventually, he will only change for worse and he wont be wrong when he will say ’ main hamesha se aisay hee tha ’