The problem is that the yarns aren't bouncing back after being stretched out. Most cotton yarns have no bounce so once you wear them and move around in it, it stretches out and gets all saggy and floppy.
Have you considered something like Cascade Fixation? It's a cotton yarn with a small percentage of elastic, and it makes the yarn keep it's shape. I LOVE it for socks and am working on a pair right now. It's easy to knit with, but you MUST do a gauge swatch to see what the finished un-stretched fabric size is.