I think your idea is brilliant!
I've lugged them around in my handbag...taking them out at the fabric store to make the button choice. Better than lugging around the sweater itself.
I've also used them to "park" my yarn needle, crochet hook, and cable needle for traveling. The swatch is usually st st, so it rolls up naturally and protects the items.