Another solution without having to start over is to buy some elastic thread, and stitch all around on the inside of the hat. I've done that before, and it worked just fine.
I turned the hat inside out, and ran my threaded sewing needle round and round, running the stitches evenly, by counting a couple stitches between each catch by the needle. Occasionally, I would turn it right side out, and pull the thread to tighten it up a bit.
When I reached the end, I knotted the end of the thread really well, and wove in the ends. Then, I turned it right side out, and it was just right.
I also use this technique to knit hats with cotton yarn.