Well I know a couple of people in PonyGAF who, if asked if they are furries, would ask yes, even though they're only fans of the art and aren't into the sexual/porn aspect of it in the least.
I guess the problems arise when someone who doesn't understand there's different... "levels" of furriness comes wandering in and goes "omg lol y'all wanna fuck ponies herp" or whatever shit it is.
Their ignorance isn't our problem *shrug*
I dunno

Personally, I don't give a shit what someone says/thinks about me. But when people take a tiny aspect of something and use that to generalise about an entire group of people it does irk me.
Double edit: Especially so with a group that seems (to me) to be as sexually closed off as much of the MLP community is. The stuck up repressed people who bitch and whine about what other people do with their character art and try keep everything cocooned in some childhood, sexless magical bubble are the ones who creep me out more.
And they're annoying as fuck because all they ever do is complain and act like insular manchildren all the fucking time.