Cotton was one of the major crops in the South during the period, and cotton mills were a major industry in the North. There was linen, too. Silk, only for the rich -- it had to be imported.
There's a great book of Civil War knitting patterns that have been rewritten with modern directions. It's published by lulu.com; author's name is DeAnn Upton. I've also seen it on Ebay.