Didn't he more or less plead with Stannis to let him take the girl with him, that a war camp was no place for a young girl, etc., and when Stannis shot that down... I don't know, I agree with those who think that dude recognized that Stannis was too far gone and when the Red Woman rode into camp without any of them in tow, it more or less confirmed his fears that he was now on his own.