Been there, done that myself! Aaargh!
I use stitch markers when I cast on more than 20 stitches.
I mark every 10 stitches. When I start my first row, I just drop them off the needle.
The stitch markers help me, and I don't have to endlessly keep counting stitches over and over to see if I have enough already!
Oh...the stories I could tell on myself...such as getting way up to the dividing of the neck...and THEN realizing I had cast on 5 too few