Slavery may have been one of the pillars supporting the Southern way of life at the time, but it was big honking pillar--the type that, when cut down, would send the whole structure crashing down.
States' rights? Free soil? Popular sovereignty in the new territories? Slaves.
The status quo antebellum was a convoluted web of sectional and political interests and conflicts, but slavery was its linchpin.