there is the possibility Bran can screw things up since he doesn't have that dude to serve as a guide.

I think the fact he said Bran wasn't ready to become him yet means changes via time traveling are possible, for better or for worse.