Same as Of Troy, the inside layer of my hats are slightly larger.
I either plan and make that the color most likely to be worn out or make the hat a little smaller and block it out.
With me it's not so much the K vs P, it's the strand of yarn that goes on my back finger doesn't get bound up as tight as the first strand I wrap for my front finger.
Even when I do socks where both sides are Ks the inside one is looser.
You don't say what the problem was. Yes it can be done.
Petroleum based knitter, removing that nasty oil from the ground one skein at a time.