Some people said that people didn't get along with Howard on the set but there's more...

"Those with intimate knowledge of the situation suggest a far more dramatic backstory: Howard was the first actor signed to the film and, on top of that, was the highest-paid. That's right: more than Gwyneth Paltrow. More than Jeff Bridges. More than Robert Downey Jr. And once the project fully came together, it was too late to renegotiate his deal. It didn't help that, according to one source, Favreau and his producers were ultimately unhappy with Howard's performance, and spent a lot of time cutting and reshooting his scenes.

...As such, when Favreau and screenwriter Justin Theroux went to map out the sequel they found themselves minimizing Howard's story line. Once Marvel learned that Favreau was thinking of curtailing the role, the studio went to the actor's agents with a new and drastically reduced offer — a number that's similar to what supporting cast members were paid for the first movie. The agents, according to sources, were so taken aback by this new figure — estimated at somewhere between a 50 and 80 percent pay cut — that they questioned it."

From Entertainment Weekly