Aren't the videos amazing? I'm using them for almost everything now!
Anyway, most yarns tell you important things on the label, such as whether or not they are machine-washable and/or colourfast (both pretty important for a baby blanket, I think, those things can be pretty messy) and what they are made of. I find it's best if you can touch the yarn you're considering, since I'm a pretty tactile person. A baby blanket should be soft, of course! I like most of my knit garments to be soft, or at least not itchy. Of course, if you're knitting something purely decorative, the texture usually doesn't matter as much.
I'm also planning on starting a baby blanket (hooded afghan, actually) and I stumbled across Lion Brand's "Pound of Love". It's pretty inexpensive for the amount of yarn you're getting, and I've read many a glowing review of the stuff. It comes in a plastic bag, but one of them had been opened at the store, so I got to feel how nice and soft it was. It also comes with a pattern for the hooded afghan I want to try printed on the inside of the label.
Of course, you could always just browse your LYS looking for yarns with "baby" in the name, since lots of brands have 'baby' varieties for just the sort of thing you're planning to do. They're also lots of fun to touch!
In general, try to think of the qualities you want in your garment when you're going out to choose your yarn. For instance, you generally don't want to use a heavy, thick yarn for knitting a tank top. Unless you're planning on having it double as a vest later, of course.
I hope that was helpful, but you probably want to wait for advice from a knitter more seasoned than myself.
Just because I'm easily distracted doesn't mean I