Okay, here's some things that could be going on. What type of yarn are you substituting? If the pattern calls for a worsted weight and you're using a sport weight, your hat will be too small. You will have to adjust your numbers like the previous posters said to do.
If your weights are the same, the pattern calls for a #4 worsted weight of one brand and you're using a #4 of another brand, you should be okay. There is a weight number on the yarn label.
But I suspect that what is happening is not you, your yarn, or your knitting but the cast on. The wrong cast on will make this hat appear too small, but there's nothing wrong with the body. Just the edge.
I knit a beautiful pair of mittens for a child. I used a long tail cast on for the cuff. Even casting on loosely, it was so tight that she couldn't get her hand in the mitten! I learned my lesson after that. For all my hats, mittens, and gloves, I now use the German Twisted cast on. This stretches as far as the yarn will allow, but quickly snaps back to shape. KH has a video on how to do it here. I think using this cast on will solve your problem. Let us know how you make out.