It is difficult to be a perfectionist and be a crafter, no? If it were me and my sweater, this situation would really grate on my nerves as well. However, I would try to relax and take a deep breath and realize that if the sleeves are the same length and no one (except you because that will be the first thing you think about when someone admires your beautiful handiwork) else sees a problem, then it's all good and you should just put the sweater together. It probably is the change in needle length that caused it. A friend of mine said that the only unforgivable mistake in knitting is splitting your yarn, the rest is just design modification. If you can't stand it, or you think it will make a difference in the finished piece, by all means redo. The only thing you don't want to do is get too frustrated and throw the thing across the room. (I have done that)
It sounds like your knitting skills are quite good. I'd love to see the finished sweater. Where did you get the pattern?