uh, yes he was (former Late Show intern Holly Hester says he even had a secret bedroom at the studio), and it is totally fair game
i suspect all this indignity about it wouldn't be present were the situations reversed and it was letterman making all the cracks about affairs leno had
doesn't matter if they're mild cracks, he's already mentioned jay's wife and home life several times - i'd say that's pretty personal
but you guys have already decided who's the white knight and who's the dragon in this whole deal, so carry on