Yeah, unless you're a loose knitter, that seems like it's going to be too tight. If you are getting 6 - 8 sts per inch, it's much much too small. If you use a size 6, that should get you the 5½ sts per inch the pattern calls for - 22 sts/4".
I don't know why you seem to want to use the size 4s with the #4/worsted weight yarn. You would normally use a size 7 to 9 with that yarn. The hat is knit denser than normal, so using a size 6 would help.
And if you do tend to knit tight (garrote the yarn) then a size 8 would probably do better. Don't pull on the yarn after you've made the new stitch and pulled it through the old one. It'll probably seem weird and loose to you, but making the next stitch will tighten up the previous one. Practice this on some other yarn or with your gauge sample. Tight stitches make it hard to get gauge and is very hard on your hands after a while; knitting should be easy and relaxing, not tense. Even stitches will actually happen when you relax your tension after a little practice, then washing the finished item evens them out further.
sue- knitting heretic