Ummmm… if he does idiotic things to irritate me and makes fun of me while I’m not on extra-friendly terms with him, then yes, I would get annoyed… and I would probably need to vent, as I’m not the confrontational type 
I would probably go to my boss or HR.
There’s this guy in my office who really annoys me with his twisted sense of humor, he’s really mean with his comments… but I just brush them off, since I don’t directly work with him. I can imagine what work would be like if he sat behind me and made snide remarks all day long shudders … and if that guys asks me to bring lunch for him, yep I would probably be all like WTH… get your lunch yourself. Please don’t judge DB, you don’t know how long this guys’ been annoying her.