If you aggravate an enderman and get into an area where it can't attack you, for example a 2 block tall roof above your head or if you stand 3 blocks above the enderman, it will stop becoming aggravated after 20-30 seconds. This is a major issue as it makes it a lot harder to kill enderman and therefore, harder to obtain enderpearls.