If you get shot, he'll go on trial saying that you "sexually assaulted" (I don't know if this counts) his wife with a dildo. Knowing the types of bullshit that juries count as self-defense, he might get off.
Did you call the cops though? At least to give them a description/license plate? I wouldn't be surprised if distinguished mentally-challenged fellows like this have some sort of warrants out on them.