You've probably forgot to do a yo, or dropped one off somewhere. It would look off if you just add stitches here and there. You don't have to start over again, but you can rip back to where you had the right number of stitches and start again from there. Then count the stitches every row or 2 to make sure you're not losing any.
sue- knitting heretic