Honestly, P5 was fairly easy (well, mostly the boss battles) but I can't say I mind that all too much. Maybe I'm getting soft in my old age.
Anyways, since we're doing series rankings (I've only played 3 through 5)
Music - 3 has my favorite, since it's more upbeat hiphop and electronic music. Not a big fan of 5's, especially the awful Mementos song
Pacing - 4 is the winner. 3's dungeon is a bit of a slog, and 5 can be pretty stifling with all the cutscenes and that damn cat
Story - 3 once again with its darker tone. 4 and 5 kept me interested, but 5 didn't feel nearly as oppressive as say 3 when everyone was getting into a stupor, or 4 when the fog slowly ate up Inaba
Dungeons - 5 obviously, since they actually have design and aren't just randomly generated corridors
S. Links - 3 is definitely my favorite, many of them were pretty poignant (the book store owners, the man with the terminal illness). 5's were a bit boring, like hearing about Ann's modeling career

or Ryuji's track team

Combat - 4 probably wins on this, a good balance between the more difficult 3 and easier 5. I also hated not being able to give orders to my teammates in 3, that got annoying real fast. Nobody in their right mind uses those status affliction spells in RPGs, so why are you doing it, Mitsuru!