Neither Mexicans nor Boricuas are known as "yellow"
That would, historically, relate to Asians.
If he were Asian, and someone called him "yellow" anything, then it would be in the same lane as "r*dskins" and other disparaging racial epithets. This ain't that.