Over the number of inches for a baby sweater it shouldnt be to much of an issue but if you want to do the maths to figure it out have a look at this
guide to gauge.
As mason said dropping a needle size may not be a problem either, unless you have already tried and know that it comes out too dense.