I started geocaching in 2002, when the closest cache to my house was 30 miles away. There are now literally hundreds within the same distance. I used a magellan sporTrak map until until x-mass 2012, when it died. It was a very simple GPS, easy to use, without a lot of frills. I had a garmin eTrex at work I played with around 2006, I hated the little joystick thing, what a pain in the butt. I used my iphone for a while. The iphone Geocaching app is pretty nice. I would suggest spending the ten bucks for it and trying out some urban caches. There are enough geocaches around towns and such, that you should get a good idea if it is for you or not. Plus - the ability to log on-the-go is pretty nice, especially during a long caching day. If you travel interstates much, there's a cache at just about every rest stop (at least in the southeast...), and looking up where one is on the phone, on the fly, is pretty cool.
Check the forums on geocaching.com. They have a "garage sale" area there where people are constantly selling GPS units. I sold an etrex (won it at a conference) for $50.
I have a delorme pn-60 now. I bought it for backpacking, but it makes a fine geocaching GPS. I found six around my area the other day. Drove a total of 80 miles, and it gave driving directions as well as walking to the cache directions. Warning about the delorme software - it has a steep learning curve, and only works on windows. BUT - you get 1:24000 topo maps of the whole country for no extra charge. Some companies maps can cost you hundreds more after you buy the GPS.
I would suggest going to REI if there is one near you. Get them to show you a few units. If you get one and hate it, their return policy is pretty awesome. Whatever you choose, make sure it is supported by geocaching.com's "send to GPS" web app. That is a really nice feature. Also, being able to do pocket queries is good as well, if you plan to go to an area on vacation, or even just drive a long way, you can grab all the caches in the area or on your route, and store them for later use.