If you are changing colors so frequently, then you should just carry the yarn along the side/edge/row change.
Finish row in 1st color. Drop color 1. Pick up color two. Twist color 2 around color 1. Knit next stitch with color 2. This can be done if you are knitting flat or in the round. Continue to twist the two colors (even when you are not changing colors for the row) each time you hit the end of the row/round and this will carry the unused color along until you need it again.