suzeeq is right. IN MOST CASES ( for the set-in sleeves) you will need to make edging around the armholes instead of the left off sleeves. Just a crab stitch won't solve a problem since a classical sweater pattern armhole usually is a little bit deeper and wider than a sleeveless one.
Crab stitch most likely will work if you are going to use it as a vest, and wear it over the thin sweater or a blouse. Other wise a strip of edging is necessary.
Good luck on your project!