If the dad is so usless that he can’t even be trusted not to harm his own child then yes divorce is the only option. I don’t see why anyone would want to retain such a lovely relationship.
But if courts are involved and dad doesn’t give up parental rights then the 4 year old is still going to get tossed back and forth. I don’t see how dad can be completely avoided. Better to keep that threat of perpetual child custody as a threat rather than letting him wash his hands of his children completely.