What should I do?

I agree, it seems like he's telling you something through his actions. Either way, back off and try to move on. I know it's hard, but the more you invest in this relationship-that seems to be going nowhere-the more you're going to get hurt in the end.