The Marvel Avengers movies are so big, I doesn't really shock me that a director used to having a lot of creative freedom/control backed out. I'm surprised that even Joss Whedon is still on board; my guess would be that he's got many stories of all the compromises he had to make to keep the job...