Because, like his background, they question his legitimacy with their very existence. His dad always gave the advice that he needs to keep his enemies close, but Ramsey has resisted that from the word go. He got Sansa in his clutches and immediately delighted in tormenting her. Northern loyalty to the Starks shouldn't be as automatic as Sansa seems to think it is, but there's no way everyone is cool flying a Flayed Man flag.

I'd say the win over him seems inevitable, but at what cost? I imagine that if they take back Winterfell, they'll find little bro hacked up cruelly in the basement, or something equally bad, just so it feels like a very qualified victory.