Years ago I stayed at the Craignair Inn on Clark island. It's an old boarding house used by workers that cut stone in a nearby quarry.
It's 70 miles southeast of Bangor but most of the drive is along the Maine coast. You will need to retrace the route to get back to Mt. Desert Island but the drive is along the coast and there are lots of little towns to stop at.
Craignair Inn at Clark Island (Maine/Spruce Head) - B&B Reviews - TripAdvisor