It may just be that the patterns are written for a certain build and you don't fit into that mold.
I find that I always have to make the armholes smaller and the sleeves thinner than in most patterns.
Maybe you can work the sweater in one size up to a point and then make the neck for the smaller size. It'll mean some math, but you might be able to finish up without adding extra.
You might want to look at a top-down raglan pattern. This way you will know from the beginning how the neck will fit and can adjust how many stitches you start with and just increase more as you work your way down.
"Trust the pattern!"
Time flies like an arrow, but fruit flies like a banana.
Hop aboard while we follow the cloud-covered van of knitting harmony, man!! Woooo!! --Hedgehog
IngridKH on Ravelry