What merigold said about being off one stitch. You write though, about using a size 6, then 5.5 and a 5... US sizes don't have a 5.5 as far as I know, so where you using mm needles? Also, isn't cotton fleece a worsted weight? That would make it thicker than the pattern in the yarn, I think. You may not be able to get gauge even if you use a 4, the yarn's too thick. So if this is a sweater, you could just choose to knit this on a 6, but use a smaller pattern size, using the size whose stitches front and back would equal the measurement you need.
sue- knitting heretic