I know this is manipulative parenting, but sometimes I tell my 3 year old that "This is your "X" sweater and in order to go to 'X" you must wear this." I've used the library, the aquarium, school, friend's house, whatever they are looking foward to.
Generally I let my kids pick their own clothes, but sometimes to get them to put on something new or different I try this approach. It's probably not that the sweater is really scratchy (especially with a shirt underneath) it's probably that it is different. Maybe if he will wear it and get a few compliments, he will enjoy it!
By the way, try it while you can, this technique doesn't work with my five year old!