He was very popular in his time because he was a dynamic brotha and he was side by side with many of the big white abolitionist like Garrison, but he was mostly into slave narratives (he spoke to a white audience for moral persuasion)
When looking at that time perion you can't ignore Martin Delaney. He worked with Douglass on his first two newspapers but he had to break away because basically what it came down to is he thought Douglass was a "fool" Delaney had thousands of blacks ready to go back to Africa and many did go but the Civil war interrupted his plans in which he served in. He did go after the war along with others.
Also many blacks was for going back to Africa in the 19th century they just didn't trust the efforts of whites in sending them their under colonization status, James Forten talks about this tremendously. Prior to the forming of the ACS, Paul Cuffe and other Liberated Afrikans sailed back to the continent under their own volition. Pan Africanism was alive in well back then, David Walker's Appeal was to Afrikans all over the world.
"We are not stunted in spirit, we are not europeans, we are not Christians that we should invent fables a child would laugh at and harden our eyes to preach them daylight and deep night as truth. We are not Arabs, we are not Muslims to fabricate a desert