If you can still get stuff in and out, I'd leave it...as Yarnlady says, I've seen them designed that way, too.
If you haven't felted it yet, could you unattach the handles and reattach them?
I did a bag recently where I did sort of the same thing (actually just lined it up wrong from the beginning)...Mine had a flap that was supposed to close with a button, so I just frogged the flap and left it with off-center handles. Voila! My first knitting design!