It's the nature of ribbing to pull in. When you do ribbing on a sweater or hat, the shape of the garment helps to stretch it out. If you don't like it the way it is, check out
this pattern. Several of us have done it and it has a modifed type of ribbing, lays flat and is very soft. Easy to make, too. The picture on the pattern doesn't do it justice.