I agree with Sue. You'll need several inches to see how the ribbing will pull in. A few rows in it still looks all stretched out and you'll think they are too big, but they may not be.
I also cast on 40 for these, but used a size US 5 needle. Worsted weight Caron Simply Soft used on these.