That's a lovely place to knit, Zina!
I've knit in coffee shops like Starbucks, doctors offices and on a bench in front of a store waiting for my daughter to show up as well as my LYS on Friday nights and always in front of the TV. I've never knit in a restaurant because that's a time I focus on whoever is there with me. I try to balance my knitting with personal attention like that.
I've never come across anyone who things it's rude or anything. Most people are very intrigued and interested in what you're doing.