Most men and support staff went with Stannis. Any trade that happened happened because of Stannis. I'm pretty sure that there wasn't a lot of farming done on Dragonstone due to it's rockiness, so there's really not anything to tie them there.