A new home in the "historic section"' looks like your best bet.
Second might be in the village of Caroline, (near LSL).
I don't agree with the statement that only old people live in the "historic section". You will make friends with people from all areas, so that shouldn't be a concern. Some people prefer the "historic section"' as it has less traffic.
PITA's can be found anywhere.
Do you best to avoid them but there are only a few.
You sound like you know what you want so go for it!
__________________
"It doesn't cost "nuttin", to be nice". 
MOM
I just want to do the right thing! Uncle Joe, (my hero).