If you actually leave it in your back pocket while sitting, the added pressure in that acute area, over the course of time and repititions, can lead to compressing the nerve (most commonly sciatic pain). Really try to take your wallet out while sitting, it's not worth risking any unneccessary nerve pinching.