If I encountered people like that I would just say,
"You know you are trespassing. The problem is not so much me as it is the man next door. I've heard that he points a gun at trespassers that walk between our houses. In fact, I seldom walk on this side of my house myself."