welcome to the world of the wonky elongated stitch. there are a few threads around dealing with this issue. here
is probably the most helpful one. i think that in the end it is just part of the stitch but you might be able to lessen the effects of it.
i did mine on a super furry eyelash yarn (before they made me violent when i used them!
) and you couldn't see that stitch. when i made it with a brushed alpaca you absolutely could and i ended up frogging! :(