How many stitches did you increase in total along the one side of the sweater? If it was more than 3 or 4 then adding three stitches in a short space on the other side is going to look odd.
If you don't want to rip the whole back I'd probably rip down six or seven inches, then reknit correctly and add any additional stitches to the forgotten side on the sixth row of the 12 row repeat.
Remember, you want to have the same number of rows in the front as the back, so if you adjust the number of back rows you'll need to do that on the front also or your seaming won't go smoothly.
The frog pond is never fun
, but it often is the best way!