I don't believe I would get mad. I might if I posted a sign saying to refrain from such things and they do it anyway, then it's trespassing. Hasn't happened, so I'd have to say no.
As for the mailman, he has an open invitation to put anything he wants in the mailbox, on the mailbox, or around the mailbox. After all, that's the purpose of the mailbox. I know he's not going to put my mail on my car. The mailman doesn't mess around on my property, he does what he's supposed to and goes about his way without touching anything he shouldn't. A novel idea, no?