I love the idea of knitting the fingers in different colors. It just adds so much interest. I would say to change colors at the base of the finger, on the back side if you are still having trouble with the holes, although you can always find some sewing thread to match and stitch it up if it is really bothersome. I found one on ravelry:
Imo if you were to change the color further up the finger it may look 'disjointed'.
I really like what you've done.