When the evokers/illusioners are attacked from far away, they wave their hands even though there are no mobs or players they could target. This action impairs the evoker's/illusioner's ability to move because they have to stand still to cast the mundane "spell", making it an even easier target for snipers like players and trader llamas.
To reproduce, spawn an evoker/illusioner, go to survival, and chuck snowballs at it from outside its detection range.