Frankly, it's okay to say no to things.
It's a disservice to yourself and anyone you say yes to if you know you don't have time, or aren't committed, or frankly, just don't want to. That's a perfectly valid reason to say no.
Will the person be mad or disappointed or whatever? Sure. Maybe. Maybe not. Who knows?
You still have the right to choose what you want.
Guilt is a terrible reason to say yes to anything.