I can't see any puffing up in the picture. For a first attempt at intarsia, though, that is very ambitious and it really does look fine. Every knitting project is a learning experience, for sure.
Next time, if you knit the right side and purl the wrong side of the color work, you'll find it, I think, to be easier and smoother. That in no way takes away from the fact that your's is a success.