Nah, you aren't doing anything wrong. You've discovered that infamous thing called "yarn puke". Where you pull from the center of the ball or skein (like it's supposedly designed to) and a huge knot of puke rolls out.
Yer sitten there for hours on end when you could be stitchen to unravel and reroll that monstrosity. Being careful all the while, to not pull it too tight so that it doesn't knot up into a knot a boyscout could be proud of.
It takes years, dahlin, years, to learn how to unsnarl that tangle of yarn sketti into something even resembling a ball of workable stuff.
But, you do get good at it after battling with skeins n skeins of the stuff and it just becomes second nature. If it's one of them there fancy schmancy yarns, I would just roll it into a ball the minute it comes outta the bag from the LYS. No one needs to be punished by trying to reroll mohair yarn puke. If it's one of them mondo huge arse skeins from like caron or red heart, ummm stick it in a ziplock baggy that's big enough to hold it, zip almost shut (here I would either staple or tape it to hold) and let the little bugger roll around to it's hearts content inside the baggy.