Pillagers that has previously spawned in the end are immune to the peaceful option. An interesting thing to note is that they will not attack the player if nearby, though returning to a harder difficulty activates aggregation.