Have you ever tried merino wool? It's baby soft, super fine wool, often used for baby knits. Next time you're at your yarn shop, take a skein and hold it on your neck, or in the crook of your elbow. That's how I determine if the yarn is too itchy or not.
Angora can certainly be itchy, especially if it's made from cut fibers instead of plucked. If they shear the rabbit, the fibers have blunt cut ends which are coarser for obvious reasons. If the rabbit is plucked, (they gently pull the loose fibers out as if you were combing a dog or cat), the fibers have a more tapered end and are softer on the skin. But sheared angora is cheaper than plucked, and is what you'll often find in commercial angora sweaters. They also aren't so careful about removing the coarser guard hairs from commercial-use angora. That's also why some balls of angora are so darn expensive... it's the goooood soft stuff.