Welcome to the forum Rorshach!
Oh man, that would really bother me. I don't know if I'd have the nerve to knit in public if I was faced with that attitude constantly! It's great to hear that the looks and jeers roll off you back.
It's men like you that will change the way people see knitting (it's about time!). I think it's simply a matter of people seeing it and getting used to seeing something they're not used to seeing. Once people see it more (which they are, slowly, there are more men knitting now, definitely), I think the strong female associations with knitting will loose their hold on people's consciousness, and the judgements will dissipate on their own.
When I put up my CafePress store, with t-shirts and buttons, etc., I'm planning a few for you guy knitters. Loof says his favorite of my t-shirt ideas is "Guy Knitter. Don't have a heart attack." (Pretty much says it all, don'tcha think? :rollseyes