When you purchased your home you were given a plot plan. this shows you what your set back his. You cannot build beyond the setback lines. This has nothing to do with the survey or the county. It all depends on what is shown on your plot plan. We were told we had plenty of room to add a golf cart garage when we purchased our home in 2006. When the time came in 2008
to do it we were told that although the survey showed one set of setbacks, the plot plan shows the actual set back and we could not build.
Don't waste your money trying to fight this. You will noy win.