RDJ was nuance, he played the prick and hero but not in an over top way. The joker was over acted.
Ooooh...
I didn't find anything distracting, gimmicky or gratuitous about the ok yes, stylized prism ledger and the character demanded itself to be in. I think you're just using 'over acted' as a simple critical shorthand to dismiss work in this mode.
Ledger had far more simplistically written impulses than what RDJ had to work with in Iron Man....
but he worked on a physical rhythm to make the character's spontaniety credible, and eloquent, in its own ugly, perverse way. especially in that resourceful interrogation scene
.. there is something raw, true twisted in those silent car window and hospital exit scenes. and just smaller moments which are easy to miss.
so yeah concluding my little 'for you consideration' ad I say there is alot of nuance in his emotional volatility buuuut...
RDJ is also loose and will deservedly ride similar waves to oscars for tropic thunder... and flipping things around a bit it might just be ledger's death which opened up the possibility for that facepainted academy recognition

Neither will win though Brolin