This is normal. There's a couple things you can do.
Sometimes a larger needle helps when doing the bind off. You can also cast on provisionally and bind off both ends so they are the same. It depends on the project whether this is a good way or not.
Or you can just do a different bind off. This one is a good one and makes it nice and stretchy. Don't use a larger needle on this one.
Jeny's Surprisingly Stretchy Bind Off