Sometimes when a person says I love you and I want to be with you but....."it will never work". They know exactly what it is that "will never work". I believe there is something that is in his mind that he feels is standing in the way. I believe he either wants you to ask him what that is or he wants you to make the first move to eliminate the problem. You don't mention where your other boyfriend is when you are spending time with your ex. Could it be that he knows you are still hooked up with him and does not want to be someone on the "side".
I think it would be in the best interest of you and him to put it all out on the table. Tell him exactly how you feel. Ask him what he wants and if there is anything you can do to make it happen.
Sometimes it is hard to forget the hurt of breaking up and the fact that you have been with another. Make sure you understand how he feels about this before you committ to trying and that you get the counseling that PDM and Carl suggested. These kinds of feelings are complicated and you would not be the first couple that needed help getting past them. If you love each other, it can be done. The last thing you want is to be in a relationship where you both are bringing up the past mistakes without help to overcome them.
