I've always thought it was ridiculous for people to be judgmental of others. Usually when someone tells me I shouldn't be doing something, I know it's just their personal preference that I don't.
If it bothers someone to see me knitting, then they aren't paying attention to what's important anyway!
"Why do you look at the speck of sawdust in your brother's eye and pay no attention to the plank in your own eye? How can you say to your brother, 'Let me take the speck out of your eye,' when all the time there is a plank in your own eye? You hypocrite, first take the plank out of your own eye, and then you will see clearly to remove the speck from your brother's eye."