No, you're not doing anything wrong. The "problem" is the switch from knit to purl and that's common even with ribbing. It may be fine once you've washed and blocked it, but there's a few things you can try.
When you do that first purl try wrapping the yarn under instead of over the needle. It twists the stitch making it tighter. Then on the back of the work when you get to that stitch you can knit it through the back loop to untwist it. This works if you're knitting flat. So if it's a cardigan you can try this.
If you're knitting a pullover in the round where you won't work on the back you can twist the stitch as I said before. Leave it twisted the twist really only shows on the back and since you are knitting in the round it'll never show. Read here where I posted on my blog about it. It's about ribbing, but you can see that it doesn't show really.
http://knittingandmore.blogspot.com/