I think it's okay as long as the baby is comfortable with grandma. He won't forget you. A week is not near enough time for that.
You need some time to reconnect as a couple and you'll be better parents for it. If you have the chance to go I think you should. I think ships have internet access plus you can call a few times. He'll be fine.