my little one has been exposed to knitting her whole life. my Mom was knitting for and in front of her since she was a baby. She understands that knitting needles are not toys and not hers. last year (4) we gave her a pair of plastic needles and some cotton and let her try (supervised.) I think because she was so used to seeing it, the "novelty" factor was low. and I don't think she felt like she had to compete with knitting for grandma's attention. (spoiled by grandma!) so she never went after the projects or tried to pull anything out.
but every kid is different.
we have "quiet time" before bed. she will lay on the bed with me while I knit or crochet and she gets to watch TV. sometimes she works on "her" project. snuggle time and knitting time in one. we also listen to books on tape this way.
maybe if there was something else to capture his attention while you worked...