Really broad question.
I think good level design really comes from a true understanding of basic game mechanics. It also hinges on how (and in which way) a game's basic mechanics allow a dev to gradually increase difficulty to make the game engaging. Kind of falls under "game design", but you can usually tell how well a game is designed by looking at the first and then a near-endgame level. Granted, following example falls under "we make a babby level or two to hook people", but look at something like sonic 1. Compare the initial stage(s):
To something later in the game...more enemies and obstacles are placed in your path...while it doesn't break the game, it fundamentally changes the feel. You end up with something like this.
While every level being full-on speed would result in a really shallow ten-minute game, looking at the two side-by-side makes it look incohesive. They really needed to find a middle ground throughout the whole game...something Sonic is still trying to get right.
Then you look at something like Super Mario Brothers, and there is a very natural progression from the first levels all the way to the end...ok, world 8 sucked, but it was still fair. Same formula, more obstacles + more enemies, but it worked. And it works as a more cohesive whole.
It also has to do with usually knowing which way to the goal. Old 2D games had it pretty easy usually, but most 3D games have pretty much caught up through guide arrows or whatever. The really well-designed games can do it without holding your hand though...or invisible walls.
---
Racing games have it easier. All you really need to do is increase the speed (or tack on horribad AI opponents, I guess). Total bias, but it's pretty easy to tell the better-designed racing games from the others, because you can actually find crazy speedruns on youtube. Using speed laps to serve my argument! Poor racing games don't really allow beautiful, perfect runs, and people usually don't post crappy runs online. Posting some XL/2097, from early, to late in the game: