Actually, I'm thinking that perhaps asking if there's a right or a wrong in the situation is maybe the wrong question.
I've found that it bothers some people if I knit when we're talking. Other people don't mind it at all.
Some people really don't like hearing swear words when we talk. Other people don't mind it at all.
Some people find it really offensive when people ask questions about their lives. Other people don't mind it at all.
Just like I don't use swear words when I know it will offend, and just like there are some people I don't ask questions about their private lives because I know it will offend, I don't knit around some people, or I knit but make it plain in as many ways I can think of that our conversation is just as important or more important.
I've found that with a lot of people, if I'm knitting while I talk to them, if the conversation turns to very serious matters, if I stop knitting and start making eye contact, and then when the conversation turns to less serious matters, and I start knitting again, they seem to mind the knitting less. Other people, while they don't say anything, seem to mind the fact that I'm knitting, whether it's distracting to them, or whether they feel I'm not paying full attention to them.
For the last two days, I've been trying to finish a pair of knitted mitts that are a present to be given to the recipient tonight before we leave town. My friends have been very happy to have me knitting during our farewell get togethers over coffee and scones, knowing that I was still taking full part in the conversations, and knowing why I was knitting even while I was getting ready to leave town for who knows how long. I really hope that I didn't offend anyone by knitting during our farewell get togethers, and was doubly careful to make sure they knew that my conversations with them were more important than the knitting.
I don't really know if any of that helps you.