Apparently he's expressed some "conservative" viewpoints, and that's why a lot of people are sour on him, i suppose.
I don't think i liked a single movie he was in (that i've watched) but he wasn't really the reason, as you point out, he's mostly fine in those roles.
I still think it's odd they'd have Mario talk, let alone with his voice.