He's a sleaze and letting him drone on about how lucky he is without copping to anything makes it seem like he's been wronged in some way. And since he hasn't been, he should just pack up his shit and go.