This is NO way to live.I have never ever recommended divorce to anyone , but in your case I think it's very jaiz.
You are young now , you can get married to someone else.You're right in saying that if he says after 10 years that its not working out , phir kya?Where will you go then?
I know it's taboo to talk about such stuff in our society , but having a physical relationship with your husband is your right.It's a basic need.
And he is only being SELFISH by asking asking you to stay with him,when he knows he cannot give you what you need,and when he's making no effort at all.Your father and your siblings are right.
Five years is a VERY LONG time.
I'd suggest you take some time off, go to your parents for a month or so, do a Istikhara, ask your father to do one for you too.
I also think this whole ordeal has effected your self-esteem when you say that you might not get some one as nice as he is.But you will inshallah.Just know that you've tried your best, you've worked hard at it and prayed hard.But that doesn't mean that you should just accept it and live a sacrificial life for him.
I'm so sorry your going through this , May Allah help you soon.Ameen