As long as you get all the parts knitted up, I don't see how it could matter. I've actually had the front pcs and the back all in progress at the same time on long circular needles. If work all around at the same time, I can be sure the length matches. I think with a complicated pattern stitch I'd probably not do it this way. If I want color changes on the same row on every piece, it works well.
You're not crazy, you're a knitter.
There's a cure for crazy.