Jan, I don't know all the levels to go through, but I certainly wouldn't follow him home yourself. He's already made an angry gesture at you. I would probably call the Village Watch, they were really on the ball with my neighbors problem. And if that didn't work I would call the Sheriff's office directly and ask for advice. They you're at least armed with "When I spoke to the sheriff, etc. . . " if you have to call some of these places back again.
Another thing I might do (but could be a woman thing) is have a friend or two follow him to his house because he'd already spotted my face! I call it a woman thing because most GF's come to each other's aide quicker than men! LOL Now, don't pounce on me guys!