Frankly, you don't have to go to that family event.
Don't just go along to get along. Because you think you can't say no or not this time or I have other plans. Because you can. Decide what you want, and do it. And be ready to own it. Live with the consequences.
Because (in case you didn't realize) you also live with the consequences of saying yes. If every year you're dealing with stress or harmful people...those are consequences too.
And guess what? You can choose something different every year.
Traditions are great until they're not.
You're an adult. You get to pick.