I never much saw the point in ponies. Children outgrow them faster than you'd think. It's better to have a horse.
Before actually buying one though, I'd recommend looking for some parks or farms that offer horseback riding lessons or let you rent a horse for a few hours. My love of horses all started when my family and I went to park that featured horse riding. There were guides there to make sure the horses stayed on the trail, but otherwise, it was a horse to ride for an hour. It was one of the best experiences I ever had.