If it is inactive, they are losing money. They are spending money processing your account and by giving you credit access. So some will drop you if you are inactive for too long.

Your credit score will drop if you are dropped (or if you cancel the card yourself) because your available credit will go down and your number of accounts will go down. From a pure numbers standpoint, it will look like you are not as worthy of credit as you were before