If you shoot any dipped arrow, including spectral arrows, at an enderman, the enderman will teleport away, but it also get the effect of the arrow, even though the enderman hasn't been hit by the arrow.