I agree with you in the context you presented.

If someone says, "that's my guy right there" or "I got a guy for____", that doesn't irk me at all.

It's when I'm at a store or whatever, and they say "What's good, my guy", though it's more annoying when it's in a social setting and not a business setting. It happens fairly often in stores or restaurants, not so much socially.

"Boss" is another one, but it's less unlikable than "my guy". I don't like those terms in general- Hon is mildly annoying to me, as are most forms of it. Honey, sweety, sweetheart, etc.

All that shit makes me mentally roll my eyes, but "my guy" in that context is the worst of the bunch, I think, with the exception of waitresses who like to add the physical contact.

None of it's really a big deal, just a superficial pet peeve.