A lot of the non-uniformity will come out in blocking. If you're using an acrylic yarn, once you've tossed it in the laundry and washed it once you'll be amazed at how much more uniform it looks. If it's wool or something else that has to be hand washed, it's a little trickier.
Fiber content might also have an impact on the issue you're talking about if you've pulled it out multiple times and restarted it. Usually it's not an issue, but if you're using something that's not terribly tightly spun, it can start to fluff up and come untwisted if you pull it out and re-use it (as I'm currently finding out is the case with Cascade Eco Duo).
And I'd second what suzeeq said... for a worsted yarn, a US 4 sounds a little small. Then again, "worsted" is a completely subjective term that covers a lot of territory. But if you've got some larger needles, you might try them and see what happens.
Finally, if you have the choice between tightening up and loosening up, go for looser. You're going to see daylight between the stitches -- especially with garter stitch -- so don't fight it. It's supposed to be that way. And don't forget to breathe. Very important. No, seriously, I mean it. If you're tense, your knitting will be too. I know it's tough to relax when you're still at the stage where you're doing this internal Snoopy dance every time a stitch actually works, but that's one of the things that's making things so tight. As a very wise knitter once told me, "it's knitting, not rocket science."