I don't think you're worried over nothing but worrying constantly doesn't do much good. It can't change the situation. Knit on! There are many other and much less productive things you could do.
Happiness is catching that dropped stitch while it's still fixable..