I think it's his voice that gets me, that, or his face... He just always seems evil.

From his character in Die Hard to doing the voice of scar in Lion King (Watched it this weekend with my 2 year old daughter for the first time) or when he played the pedophile in Lolita... Just always end up not liking him...