Do you ever get the "Oh, I'd love to do that, but I don't have the time." This always rubs me the wrong way, although I don't know why. I am a busy person myself, but knitting is something I love to do, so I find time when I can.
It seems to me that people find time to do things they WANT to do. The people who say this are those who spend their leisure time doing all sorts of things (reading, golfing, boating, gardening, watching tv, whatever).
I always (politely) say, "Of course you have time, you just prefer to spend it doing something else."
So good for you on speaking up!! Also, strike that person off your knitting "to do" list!