I know exactly what you mean. If you haven't noticed garter stitch looks different on both sides of the needle. You are always knitting on the side that looks like purls so what you need to do is pick up and knit the stitches from the side that looks like purls. If you've already seamed the shoulders and upper arms then you can knit in the round from the inside of the circle or I think you can pick up and purl from the outside then knit the next row, etc.
Here's a picture I took of what the stitches need to look like if you are picking up and knitting. I made sure that I had that side on the outside of the sweater when I did mine.
ETA: I see you said purling didn't look right. Then I'd probably work the inside of the circle if you have it already seamed. Next time you'll know.