I had dinner last night on the back deck of a Villa which is on a golf course. It was near dusk and the number of golfers was dwindling, but I still found it annoying to have the golf carts passing within a few feet of where I was sitting. I'm told it it much worse through the day.
But, to the point of your post, I have met people here in TV who have sold their homes abutting golf courses because of the kind of thing that you describe. It is poor golf etiquette, and poor manners too.
We met one lady who collects the golf balls that land in her yard. She said that in a ten year period she had collected 1000 balls.
I believe that if I were in your place, every time I saw a golf ball in the yard I would immediately go out and pick it up and throw it into the street. I wouldn't be bothered by having the golfer argue that he wants his ball back (and I would toss it where it would be a difficult lie for his next shot.)