Yes, I fear that not inviting them would cause even more problems for you.
Would there be plenty of your friends and relatives at the party?
Would it be possible to just stay away from them and chat t people you get on with?
I know it wouldn't be much fun for you, but it might be an answer.
Couldn't you arrange a special event for you and your husband ~ as well or instead ~ like a romantic weekend away, perhaps?
