Steps to Reproduce:

1. Create some XP orbs by using Bottles o' Enchanting or killing mobs;
2. Try to get close enough for them to start slowly moving towards you;
3. Quickly switch to spectator through the "pause" menu or a command.

Observed Results:
The XP orb doesn't stop targetting you after you switch to spectator.

Expected Results:
The XP orb instantly loses its target.

Notes: This is more easily observable with the use of command blocks to keep the orb in place while still targetting you.