First off, stitch holders are only for that--holding stitches. When it comes time to knit them, you need to put your stitches back on the needles.
It appears that maybe you shouldn't have bound off the stitches. The pattern should have been clearer. It makes no sense to have one sleeve bound off and not the other.
I would 'un'bind off the stitches so you can work across as the pattern says.
"Trust the pattern!"
Time flies like an arrow, but fruit flies like a banana.
Hop aboard while we follow the cloud-covered van of knitting harmony, man!! Woooo!! --Hedgehog
IngridKH on Ravelry