Do they seem like they want
to fall out? Latch hook rugs don't fall apart, I have to wonder if your ears would (great idea, btw.)
If you used wool, I really don't think I'd worry about it since they will kind of 'felt' to each other when the fibers are in contact for a long time.
Even with acrylic, I think they'll probably stay in place on their own.
If you wash it, do it by hand rather than in a machine; or if you do use the machine, put the hat in a bag.
Worse case, if a piece should fall out, you can just put it back in.