i know it as an insult (a non verbal 'F**c or P**s - off') when it is the V with the back of your hand showing and as peace or victory with the front of the hand showing.
how seriously it is taken depends on who is doing it i think. you often see it being done between freinds in jokey manner when they are in a situation were they can't talk (say if i was on the phone to my mum and my mate started tickling me i might gesture the back-of-hand V at him while trying not to giggle).
thats not to say it doesnt get used in a more aggressive way too, just like other forms of swearing.
I am Tarriknits on Ravelry
This is my not necassarily knitting related blog