Yes, finishing knitting that was done in pieces or not made in the round involves some sewing
. Not my favourite part, but necessary to show off all that great knitting!
Different techniques are used for different reasons, mattress stitch might be the one you want for a vertical side seam on a purse.
There are some videos right here at knittinghelp (scroll about halfway down the page to FINISHING):
Here are some good pictures of seaming 'in progress' at knitty.com:
As for picking up sts after binding off, this is done for various reasons. Here is an explanation & more photos from knitty.com: