Thanks for all your responses. I had very nearly talked myself into frogging, washing the yarn, and starting over again.
Then I walked back out to the other room and took another look at the sweater, showed it to DH (who was sleepy and didn't really pay much attention), and I realized -- I CAN'T frog all my work!! Even with those little details that bug me, I'm still proud of my efforts. It's the first sweater I ever started knitting (although I've finished a different one in the interim), and I do want to finish it. Maybe there's a few parts that I can redo, but I don't think it's necessary to start all over from scratch. I can definitely see myself wearing this sweater once it's finished.
I can be an uber perfectionist at times, and a lot of it has to do with my current mood. That's why I knew I had to wait a few days and think it over, change my perspective, and hear some other voices (besides all the ones in my own head!) before making a final decision.
Anyway, I do appreciate your kind comments and encouragement; it's the reason I love KH, and there's no other place on the internet quite like it.