That's the one thing that drives me crazy about the Noro yarns. For some things, the change in color didn't matter, but for a sweater that I'm working on now, I wanted the sleeves to be close to matching each other. I went to great lengths to make sure to start the skeins at the same color, then *KNOT* different color totally.
You can certainly take out whatever colors you like and switch them around. The Silk Garden felts together very well, so joins don't need knots.
There is a mustardy color that I find unpleasant in one of the Silk Gardens that I used, so I just took it out as I went.