Welcome to the wonderful world of knitting and KH! I hate to tell you this, but knitting isn't cheap--especially since your goal is to a) buy yarn, b) use it up in your project, and c) repeat the entire process an infinite number of times.
Of course, you can do what I often do: Reuse the same yarn over and over and over and over and over again because I'm not happy with how my projects are coming along. Now that requires a lot less money (but your yarn starts getting warn out eventually).
Yeah, I agree with Jan--you're accidentally picking up part of the other stitch when you pull the working yarn through it. I bet if you look closely, you'll see that the fabric you've created is very tight in those places because you essentially have two rows knit together.