Only about 5% of the people in the South actually had slaves .

During the far majority of the many years of slavery on this continent it existed with the approval and encouragement of the British Monarchy .
Do the math . We were not even a Nation until 1789 . The Federal Gvt was very weak by design until Lincoln . The States were superior to the Federal Gvt .
The Fed Gvt was still fighting off the Brits until 1814 . Yes it took another 51 years for slavery to end which was sad but the Fed Gvt had very limited power to do anything .