One idea would be to line the hat with polar fleece. Here is an article by Techknitter about doing that, and the first reason she given for doing it is to make the hat smaller.
LINK
You mentioned crochet a couple of times. It might work to crochet around the edge for a row or 2-3 with a small hook and maybe space your stitches out a bit too. You could try it.