Strooper, I likewise think it's awesome that you knit! As was said earlier, knitting used to be an all male practice at one point anyway.
I think you guys are right about the taunting from siblings. They are just jealous. In fact, my older sister has already found five (FIVE!) scarf patterns on ravelry for me to knit her, lol. She wants me to teach her and my niece this Summer. So now he'll be outnumbered!