Double Knit Hat

[RoseFairy]

So, I decided I wanted to try knitting a double knit earflap hat. Problem, my tension is really awful. When I knit the earflaps it's fine since I'm knitting it flat, but once I switch to the round it's ugly. I know it's because my purls are looser then my knits, since it's the purled inside that's loose.

Anyone have any tips on how to even out the tension for the inside to match the outside better? I'm not really working from a pattern, I'm just following a basic earflap hat pattern and adjusting it for my double knit hat. Also I only know how to knit English/throwing style, in case that's important.