The official line is that you’re just repeatedly, respectfully “busy.” You and your husband need to stick to this in a united front. (He isn’t allowed to leave the get-together-possibility door hanging open for you to always be the one to slam.)
Considering it’s not in your husband’s best interest for his good friend to know that you think he married someone horrible, you’ve got to let them come to their own conclusions.
But if it’s just their attitudes you can’t stand, and they’re not truly posing a threat to your position or your organization, then this is a classic case of how to get along with difficult co-workers. Be civil and respectful, but stand firm in your boundaries, don’t engage more than you need to, and never give them ammunition to use against you. And perhaps you can do some good for all of you by setting an example they can learn from.