Anthony was racist (pretty openly) even when the show was popular. Patrice and him going at it hard while somehow still managing to stay cool made for some good radio.
Without Patrice, it just turned into right wing radio rants by Anthony. The show didn't last long after that.