When invisible, endermen can see you when you stare at them, causing them to attack.

Why it's weird: Skeletons, spiders, and other mobs (temporarily) won't attack you if you are invisible, but staring at endermen, that's a different story.