I have, on multiple occasions, been attacked by endermen without provoking them, in all areas of the game. The two most damning situation in which this occurred involved me 1.) staring at a wall in the nether, and 2.) staring down into the abyss in the end. In situation 1, I was walking through a small tunnel in the nether, climbing an incline with nothing but raised ground in front of me (netherrack), when two different endermen teleported into the tunnel I was in at the same time, already angry and screaming, and attacked me till I died. In situation 2, I was surveying the size of a very large gap beween two islands (the other island way too far away for me to provoke any endermen that could have been on it) when an enderman started screaming behind me, and started attacking me. 

I have experienced this issue in 1.16.100, and previously in 1.16.40 and 1.16.100 Beta.