First, you're not the only one to make that kind of mistake.
I have done it and had to frog many projects.
I try to read through every pattern at least once and then circle or highlight the numbers for the size I'm working on so there is no confusion.
That said, I don't think you have any other choice but to rip down to before your decreases and start again. If you feel like you can do it you can just frog it to the first decrease row and pick up your stitches or else you can take it down a stitch at a time till you are back to your first decrease row. Then just start again.
If you need help picking up any dropped stitches there's a video in the Basic Techniques sections that will help with that.
If you are getting your gauge or are close to it and you are that far off with your stitch count, your sweater will not fit.
You will be so much happier with your sweater if you take the time to make it right!