There is a book called The Gift of Fear, by Gavin DeBecker that I read a year or so ago that talks about "funny feelings". I would definitely not invite some strange dude over to my house, especially if I was skeeved out already. We are taught to be so non-discriminatory and PC that we ignore our basic instincts. Go with your gut, especially since you said the response was pushy.
No, I am not ashamed that I have a major crush on a man who dressed in tights and made faces for most of the 80's, and now plays a crippled drug addicted doctor...and is almost the same age as my father.