If Trump won the nomination but got completely beaten in the election, wouldn't that cause some major shifts in the republic party?
Changing this question from "Trump" to "any candidate with extreme positions/rhetoric," the answer is maybe but probably not.
Parties will shift to deal with changing demographics or popular attitudes about stuff, but most politicians in a party care primarily about their winning in their own districts, so there can be a conflict between local an national incentives. The DLC was formed to move the Democratic Party to the center by Dem politicians who felt like the party's image was hurting their chances. I don't see a feeling like that in the GOP yet.
After Romney lost, parts of the GOP leadership suggested dropping or softening opposition to gay rights and to immigration, since those were hurting them with growing parts of the electorate. The immigration thing brought a big backlash (Rubio tried to put together a bipartisan reform bill and now has to campaign against it). They didn't really do anything to change on gay marriage, but the issue was settled by the Supreme Court and they mostly seem happy to not talk about it.
Basically, the most obvious ways of attracting new voters involve alienating big chunks of the people who already vote for them. Not so much a problem of the GOP as a reflection of where the country is right now socially.