Yeah, eBay is the best place. I picked up a sweeeeet Best of Jimmy Smith 12" from there for around $3 a little while back.
Otherwise, try thrift stores, and I know a few record stores in my area have second hand vinyl sections - there's even a smattering of new stuff in some stores, but you'll be paying through the nose for that.