Go and pick up the recently-released U.S. version of KOF '98 Ultimate Match. I think it's only $19.99, and it is hands-down the best KOF game to date. '98 to me is still the high-point of the series, building on the already-awesome KOF '97 and making for what is essentially the perfect 2D fighter. It's not balanced, but it's also far from broken. A lot of people feel that '98 Ultimate Match has done a pretty good rebalance job; while some characters like Wolfgang Krauser are clearly at the top, you can do well with everyone. KOF XII looks like it's following SF4 in that they're going back to the older games (so '94-'98) in terms of character design and while I'm still not sure, move sets. If you take out Ash Crimson (metrosexual "Guile") and Shen Woo (metrosexual brawler), it's all characters from the first few KOF titles, plus Raiden who is from Fatal Fury.
Let me break the games down based on release dates; the spaces are separating the games with different play styles.
KOF '94- The original. It feels dated and old-school "SF2" now. Introduces the basic team play system; you pick from teams of three characters and play elimination-style matches. You are unable to do team edit; compared to its sequel, it almost feels unfinished in terms of character move sets. You can also have your team mates jump in from the background to assist you under certain conditions. This was rarely used by players and I honestly can't even remember when it was removed.

KOF '95- Builds on the original by adding team edit, more moves, and an overall better game play system. Still feels dated, but very playable. Both '94 and '95 also allow characters to hop forward or backward, manually charge super meters (which when full, slowly deplete), and dodge attacks by hitting AB.
KOF '96- Completely re-does everything; new characters, new sprites, changed moveset attributes (a number of characters no longer have projectiles that go full screen); now you can run and AB rolls.
KOF '97- Builds on '96. Adds more characters and moves. Brings back the '94/'95 movement/super system. The '96/'97 style is called "Advanced," the '94/'95 style is called "Extra."
KOF '98- The ultimate old-school KOF. Brings back a large number of characters, adds moves, tweaks, and generally just fucking rocks. Nothing new is introduced; it's more like everything that was awesome about '94-'97 rolled into one complete package. It is only missing a few characters from '94-'97. A number of characters (mostly from Fatal Fury and Art Of Fighting) also get "Another" versions, which have different movesets and restore the full-screen projectiles.
KOF '99- Completely re-does the game play system again. Introduces the Striker System, where you can call in a character for an assist attack (this cannot be done at any time; you must have it available; IIRC each match starts with the Striker meter full and allowing for four assists). New characters no longer roll, but dash through attacks/projectiles. Team size increased to four characters; one must be selected as the striker and he/she cannot be used as a playable character for that match. A number of older characters got changed-up movesets.
KOF 2000- Builds on '99; removes the striker meter being limited, making it more like a Capcom Vs. game in that Strikers can come out almost any time. -Last KOF developed by the original SNK-
KOF 2001- "Builds" on 2000 and makes it so that you can now arrange teams to have up to THREE strikers and call them out all at once if you want. Developed by Korean company Eolith; KOF 2001 is probably the most-hated in the series. Fans greatly disliked the character changes, art style (it's putrid and lacking color), and music. Introduces some new game play like the "wire system," which allows for limited juggles in the corner, IIRC.
KOF 2002- Eolith redeems itself with 2002, which could very well be titled "KOF '98 Two." The game drops the striker system and goes back to the original three-on-three team system, yet keeps all of the changes introduced since '99. Also like '98, there are no new characters, just old characters brought back.
KOF 2003- Development shifts back to SNK, now called SNK Playmore. Introduces a new shift system, which lets characters tag in and out (like a Vs. game) when the tag meter is full. A number of characters are changed up and/or redrawn. Last Neo-Geo KOF title.
KOF Neowave- The first "new" KOF game to be released on the Atomiswave platform. Developed by Sammy/SNK, this game is essentially another version of KOF 2002, using the same sprites on new backgrounds, but varying the roster slightly by adding some characters and removing others. Notable for adding in three different game play styles, including one with parrying. Not as well received as other games in the end; most people still prefer KOF 2002 over it. The Japanese-only PS2 version got two extra Eolith characters (Angel and May-Lee), who were removed from the Xbox port.
KOF '94 ReBout- 10-year anniversary re-release of KOF '94 for the Playstation 2. The game plays the same as the original, but team edit is available, two additional characters are added/playable (Saishyu from '95 and Rugal), and the the sprites have been cleaned up and filtered so that everything runs in 480p. Did not come out in the U.S.
KOF XI- Released more than two years after 2003, this game goes back to the tag system that it started and refines it, allowing for tagging at virtually any time, plus some more new game play tweaks and additions. The arcade version shook up the roster by adding in a number of new and/or never-before-seen in KOF SNK characters and removing some long-time staples, but the PS2 version brought them back (ripping them right out of NeoGeo Battle Coliseum) as unlockable extras.
KOF '98 Ultimate Match- The Ultimate KOF. Takes '98 and adds in the "missing" characters from '94-'97 that did not appear in the original (even the bosses), adds new moves, new backgrounds (and older backgrounds and BGMs from '94-'97), tweaks the balance, changes the Another versions (they feel more like full-on characters now), and just generally fucking rocks your face off. Feels like a "best of" game for '94-'98. Counting the Another versions, there's a whopping 64 playable characters.
KOF 2002 Unlimited Match-Follows what '98 Ultimate Match did and acts as a "best of" game for '99-2002. The play style is the same as 2002, but every single character from the games mentioned appears, new attacks have been added, and new stages/music has been created. There are a few alternate version characters and unlockable playable bosses; counting everyone there are 66 playable characters!

Also there's three 3D KOF games:
KOF Maximum Impact- KOF makes the move to 3D. The game is primarily 2.5D but you can sidestep; plays like a cross between Tekken and the 2D KOFs in that there are canned combos, but special and super moves are still there in addition to normal combos. The game feels broken; certain characters in particular can abuse a downed player and keep hitting them until they are dead if they don't do a recovery roll in time.
KOF Maximum Impact 2- Released in the U.S. as "KOF 2006" for whatever reason. Fixes a lot of the issues the original game had, and adds in a lot of characters. You can now parry attacks. Still feels a little a slow and has some slowdown issues. Huge challenge mode and story mode for every character available.
KOF Maximum Impact Regulation A- The first in the series to get an arcade release, this upgrade to MI2 adds one new character, and brings in Blue Mary, Ash Crimson, and surprisingly Makoto Mizoguchi from Fighters' History. Plays faster, feels better, and doesn't have slowdown issues. Team Play is now the primary play mode, but the story and challenge modes are gone. Did not get a U.S. release.

You're likely going to find that people will tend to favor either '98 or 2002 (which also just got the UM treatment; 2002 Unlimited Match hit Japan last month) as the best games in the series. If you want to check out everything, you can play '94-2003 emulated, and the rest of the games are all on consoles as either stand-alone or collection releases. (Every single KOF is on PS2 now.)
Some people might tell you to check out some games because of the storylines; that's crap, I love me some KOF but the "storylines" are ridiculous nonsense-animu drivel and not worth your time. (Who the hell plays fighting games for their stories, anyway?) I think if you want to play the best and have the best variety, I'd get KOF '98 UM and KOF XI, play 2002 (and get the UM version if it comes out Stateside) and wait for XII.
I'd also like to recommend NeoGeo Battle Coliseum. It's not KOF, but feels similar to KOF XI (same hardware); it's basically SNK Vs. Capcom SvC Chaos DONE RIGHT; the game builds on its game play system and adds two-on-two tag play matches, bringing together characters from Fatal Fury, Art Of Fighting, KOF, Samurai Shodown, Last Blade, World Heroes, ADK, and more. The game has some pretty bad load times on PS2, but loads quick if you play it on a PS3. I also heard that the U.S. version removed progressive scan support for whatever stupid reason, but it looks great upscaled on PS3 anyway.
