Thanks for the suggestions!
The holes are right below the cast off edge and appear where the decreases are. I think maybe my gauge was too loose. The yarn is stretchy, but I made an effort to cast off loosely so the neckline wouldn't pull. I think I messed it up the other way by having it too loose.
I think I could pick up stitches below the neck (because the holes are right on the edge). I'll definitely try to cast off a little more tightly next time. I honestly have no intention of taking the sweater completely apart and reknitting (I've seamed all the edges, sleeves, etc).
Any suggestions for what kind of border to put on? I'm a new crocheter (this sweater!). And yes, it's really nice to have those skills. I guess this is my chance to improve them.