Yup, needle size is key... And the gage you create with a yarn/needle combo has lots of affect on the fabric you knit up. For instance, a smaller needle will create a tighter gage (more stitches per inch) and create a denser fabric. So, it will have less drape. Think of drape like the differance between silk and canvas; the silk that, say, night gowns are made of have have lots of drape, and canvas, that a sail is made from, has little drape and is stiffer.
So, a larger needle will create a more open fabric with more drape, and less stitches per inch.
This counts a lot when you are figureing how you want your fabric to work. If you made a pot holder, you would probably want to scale down the needle size you use and have it be smaller than the reccomended size, so you would have a smaller gage, and less drape. This would allow the fabric to be "stiffer" and thicker which would be very helpful in not burning your hand through the fabric. If you were making a light summer top that you would wear a camisol under, you might wanter a larger needle, to create a larger gage with more drape. This would create a lighter, less dense fabric that may allow more to be seen between the stitches (thus the camisol!).
I knit my socks on size 3 needles, instead of the reccomended size 9 needles, to create a much more dense, durable fabric with little drape so my socks have a tightly knit fabric that doesn't slouch.
However, I knit my tester sweater (didn't use a patern and just wanted to see what I could do intuiting my way through) that was worked in thick yarn in size 15 needles, instead of size 11, so the sweater would be less bulky and have more drape. You can see through the fabric when it's stretched a bit though.