See lot and house go hand in hand. Have to love the house and the lot has to be situated just right. You don't have to settle. Surely there will be more lots opening that will accomadate the house you want to build. You can put your agent on alert to grab one or hold one if it opens up. We got a spec but we looked at 15 houses and took it down to two. Both fit the bill but the one we chose was roomier and set nice on the lot. Building is a pain because of the issue you are running into. And no...any house on a wall was ruled out. Don't like being walled in
