I've also never had anyone react negatively to my knitting. During the greeting people come up to me and ooh and ahh over whatever I'm making. Now that I knit I can pretty much recite the main parts of the sermon. Don't get me wrong, I don't knit during prayer or when we are reading from the bible. Also my church is a very interactive, more like school, church. My pastor gives individuals homework assignments to be presented the next Sunday. I guess that's why I'm so comfortable knitting there because we aren't the sit and be quiet kind of church.
I believe if you think God would think it's inappropriate for you to knit during the service, you shouldn't do it. But I think He doesn't mind me doing it.