I'm the organist at my church, and I knit during the sermon every Sunday. No one has ever complained, and I'm sitting right behind the pastor, on the front row of the choir loft. There is a low wall in front of my seat, so no one can really see what I'm doing.
I've been a church organist for almost 30 years now, and I've heard lots of sermons. I have a good, strong relationship with the Lord.
I'm usually working on a charity project during the sermon, so it almost feels like the item is being "blessed". I sometimes pray for the person who will be receiving the item, so that makes it even more special.