If you worked across the front of the sweater, cast off for the armhole, worked across the back of the sweater and cast off for the armhole, you should be back at the edge of the front with 120 stitches.
At this point you should be working back and forth on 120 of the stitches.
Did you have the right number of stitches to start?
Tell us what you have all the way around. How many stitches in front, how many in back, how many CO for the armhole.