Its not irrelevant, its how you are perceiving him to be, that does matter.
Let me ask a question. Five women wearing dresses are standing together. I'm asked which dress I like the best and I respond by saying, the one that the black girl is wearing, would that be racist?