Well, this will depend on...
1) How tight/loose of a knitter you are. (and also the pattern)
Tight knitting/patterns won't have as much give so you may want to make a slightly larger size.
Loose kniting/patterns will have more give/stretch.
2) How true/accurate your gauge is.
If your gauge is taking a wee more to make 4" then you'll probably need to make a larger size.
If you're making 4" with less than make the smaller size.
3) What material yarn you use.
Wool and cotton won't stretch much.
Acrylic can stretch a lot.
4) What size the rest of you is.
If you have a larger bust, but smaller waist/hips, you can always create some shaping by making the smaller size for the waist/hips and then adding sticthes to get to the larger size for the bust.
More:
Having looked at the pattern...
1) Garter stitch tends to have less give than say stockinette, so make the larger
4) Given that this cardigan is open on top and flares at the bottom, I would make the larger size on top and then consider adjusting down a bit on the bottom half (if your waist/hips are smaller).
