You are right about where the Buffalo herd was on El Camino. It was where they later built Freedom Point. And I can remember taking our visitors there to a parking area, where you could feed them carrots over the fence and take photos. And the story that I kept hearing (can't confirm if it was true) is that Florida taxes have a favorable rate if the land is used for livestock.
So (as the story goes) The Villages would keep Buffalo on land that they didn't need right away for the tax benefits. Plus it was a great for marketing - Give me a HOME where the Buffalo roam (which is how the high school got it's mascot name). As they eventually developed those areas they would keep moving them to more southern areas until those areas were ready for development. Then (as the story goes) when the buffalo were in the southern part of The Villages, someone wanted a close up photo and they climbed over the fence. Didn't hear whether they got injured, but The Villages feared potential law suits and either got rid of, or moved the buffalo out of sight (probably to land they they don't need to develop soon and receive the livestock tax break).
Anyway, that is the story as I heard it when we first starting visiting The Villages back in 2002. I can also remember reading a story where one buffalo got through a fence, and that might have also resulted in there disappearance. But I'm sure it wasn't the buffalo hunters shooting the herd for meat to serve in The Villages restaurants.
Although, at some of the restaurants you never can tell.