The problem was Duchovny lingering around so long, and the "mythology" never being put to bed. It didn't give Robert Patrick and Annabeth Gish much time to shine. There should have been a clean finish to that storyline (and now you're never going to get one), and Gillian Anderson could have stayed on in a lesser capacity to ease the transition. I tend to avoid the mythology episodes, so for me the show maintained a certain quality until the end.
Oh, and I would do terrible things to Annabeth Gish.