Our neighbor does NOT walk up our driveway to his fence nor does our driveway lead to anything other than our garage. That being said, we have a corner courtyard, so I believe that may be different with the driveway set up.
Neighbors are usually nice and respectful. We have never had a problem.