The slouch looks okay, I think just the ribbing may be too loose. You can either cut it off and pick up the hat sts to redo it from the top down on smaller needles and bind off the edge, or you can run elastic thread through the sts on the inside.
If you think the whole thing is too big, then rip it out and reknit on smaller needles, less rounds to make it shorter, etc. The st repeat is over 9 sts, so you could CO 9 or 18 sts less if a smaller needle makes the sts too tight.
WoW! It looks incredible on you! You look great!! I hope you can get it fixed. I was going to suggest sewing a headband inside the ribbing. Some sort of fleece or knit material but I think elastic will work great too. Good luck!
Late at night, when the family's all asleep,
you will find me counting stitches
instead of counting sheep!