I completely agree here. He should respect your wishes. Maybe he just can't relate to loving knitting and taking it seriously. (I can't relate to loving brick-laying, but I could imagine someone taking it seriously as an artform, and I would definitely respect their wishes if they said to lay off the tools!) In any case, it's something that's important to your heart, and there are natural limits to what feels right when someone's joking about something important to you. Trust your heart.
When Sheldon crosses that line, which I think is natural if someone can't relate to your passion, I stay firm. I always try to communicate clearly, with understanding where he's coming from; like, I just tell him what I want, without anger in my voice, but clearly. Of course sometimes, if it's really
crossing a boundry, I can't help getting mad. That's healthy, I think, and can effectively communicate how important something is to you! Definitely hold your ground, and communicate with as much understanding as you can, and do forgive him for not knowing; but don't second guess your feelings, it's clearly important to you and he should respect it, "period," as anise said!
I used to have no idea that I was crossing a border with Sheldon when I'd constantly offer "helpful" comments when he was doing something. He'd feel disrespected, like I didn't trust him to do things on his own. Now I totally get it, but it took me a little while. Now I can offer comments here and there, and he doesn't take it that way, because we've established that I trust and respect him. But first we had to communicate, and it was a little uptight, and that was just part of the process. Now we can totally joke about it.
Definitely you're on the right track I'd say, and not over-reacting.