YES! Knit with what the pattern will mostly be in.
is it a sweater? They are probablly having you knit the ribbing with the smaller needles, and then switch to the larger for the body. That way the ribbing is niiiiice and gathered.
Does it say anything in the pattern like "X stitches and Y rows =4 inches with larger needles" ? That will tell you for sure...