if you have a 'suit' meant to be worn casually, sure I guess, but otherwise you look kinda goofy imo. personally i've never done it, and there are so many 'casual' derbies (color, material, finishing) out there nowadays that I don't think it needs to be a thing. also rubber soled fashionista loafers > sneakers.