Hi Sara,
I wouldn't ... depending on how many rows of each color you're doing...
No need to make more finishing work for yourself...
If you're switching colors every 2 or 3 rows, just carry the colors up & start using a new one when you need to. Be sure to occasionally "catch" the colors you're carrying (by twisting the one you're carrying around the one you're working with) so there aren't big loopy carrys on the back that the wearer would snag a finger in.
HTH
Diane