If I'm understanding your post correctly, you've only just done the ribbing? If so, then you're probably okay-- the ribbing will pull in and be more narrow than the stockinette st part. But if you've done the st st part for a few inches (vertical inches, that is) and measured across and are 4" short, then that's a different story.
I know this isn't the answer you want, but if it were coming out way too small, I personally would just rip it out and start over. The side panel thing is going to get tricky with the armhole shaping, and unless you are an experienced knitter or dressmaker and know how to shape it, it's probably not going to come out well.
And yes, it sounds as if you did the right thing in getting gauge, that is, trying the different needle sizes. When working a gauge swatch, the bigger the better. If it says 18 sts = 4 inches, I would cast on 30 and work for 20 rows-- you'll really get a sense of how it's going to come out, especially until you've been doing it for years and get to know how you knit.