You are a better knitter than I to even attempt a sweater. There are 2 things in life I expect to never do - run a marathon and knit a sweater.
Anyway...do you have a LYS you can take it to and have them watch you knit and see if they can see where you are making the error? I'm sure I abuse my lovely LYS owner, but I try to make it up to her in purchases and cookies : ) I know you have a great little shop over towards Albany off New Karner Road....I visit them everytime I am in town and even got my Red Heart mother shopping there now...Trumpet Hill I think.
Best of Luck !!