Well, speaking solely for myself...
I've always loved animation since I was a kid. But living in America, 99.9% of it was watered down, innocuous tripe. Heck, for the longest time, most American cartoon characters weren't allowed to even throw punches, til something like Batman: TAS came along. Of course, i didn't really think too much of that when I was younger. But that line of thinking changed some time in 97-98, when I had my first experience with anime. I don't know what the show was called, but It was this sorta Gundam- like where a post apocalyptic city was being ravaged by this giant air craft or something (lol yeah that narrows things down). But what struck me was this one scene where 2 guys who were having a conversation about religion or some shit, and then one of them just comes out of nowhere and shoots the other guy dead, lying in a pool of blood. For a kid who's been used to watching campy stuff like Spiderman (from the 90s), my mind was friggin blown. It wasn't just the violence, but the style, the atmosphere, it was definitely something that was appealing. I wanted to see more shows like it, and since American animation weren't going to provide, more anime was the only alternative.
I suppose you could also say that anime was also a justification for many of us who liked cartoons but now had an excuse for when people would try and mock cartoons (hey, look there's blood and swearing, so you can't say nothin!). Cartoons for grownups, as it were. But even then, it's not anything surprising. We all try and justify the stuff we like. And I don't think anime fans are worse in this case than anyone else. And like most other hobbies and whatnot, you will always find a small contingent that makes every one else look bad i.e. trekkies, Star Wars Geeks, etc.
I'm a bit tired so forgive me if this post isn't as coherent as I'd like it to be.