I never leave the house without some sort of project if I don't already have one in my car. I find most people pretend they don't even notice what I'm doing. The only direct comments I've ever gotten were from crochet-ers telling me how they wish they could knit. Which is funny because most knitters I know also crochet.
My worst experience happened when I forgot to lock my car when I was running errands. Someone stole my favorite knitting bag (had a picture of yarn balls on it and said "ball sack" on the side) and the project within.
Dunno why anyone would want to steal an unfinished knitting project