When I knit years ago, I didn't pay much attention to gauge, either. Since I made mostly kids' things it didn't matter. Looking back, though, most of the adult sweaters I attempted were, for some odd reason, too small. :rollseyes:
That change in needle should do it, but as they say. . . gauge swatch!
I don't know anyone who likes to do them, and I must admit I rarely make one a full 4 inches long.