Honestly, I think the part was originally written with a Black person in mind, and QT being QT, decided to just cast himself because a) he thought he could pull it off, b) he wanted to be controversial, or c) he's clueless that people might find it offensive. I also think he figured he could justify it, because the character's wife was Black, and therefore Jimmy couldn't be racist. Not that that makes any sense.

I've always found it the most offensive line of dialouge in his movies, and it annoys me that line is in a film a REALLY like.