People who "can't" wear orange just haven't found the right shade yet!
I'm a red head and people are always telling me how red heads shouldn't wear orange or red. Odd. I wear those two colors all the time!
It's a great sweater. I can see it layered with a soft shirt underneath it. Then the neckline will look like you meant to make it that way! :D