I have noticed this issue a while ago. Every once in a while I have to repair my automatic sugar cane farm on my server because the enderman spawns inside and picks up dirt. It breaks the plants and then I wonder why I get no drops again. I run completely vanilla server with no mods, plugins, or data packs. I placed a few boats to verify if it was an enderman or another player, and sure enough I got my answer.

Spawn chunks, gamerules: doInsomnia false.

Describe the bug you're experiencing here

Endermen should ignore dirt or sand with sugar cane or cactus above, as it goes against the Mojang principle of avoiding mob interactions that were not caused by the player.

Please do not mark as "feature request" and let this report stay.