Berbers(and some other native groups) were there before the Arab invasions and consequent Arab empire. Also France brought like 1 million French settlers to Algeria during colonization and I'm not sure on the number for the rest of the North African nations, but I know it was quite a lot. So i guess it all depends on what ethnicity you're talking about. In all honesty race is a bullshit category when it comes to stuff like this because there are so many "grey" areas.