XPOrbs are attracted to me on spectator mode. Since Mojang fixed the flames appearing around spectators in lava I know they care about making spectators as invisible as possible to the outside world.

EDIT: Further playtesting seems to indicate that it happens to xp orbs that are attracted to you before switching to spectator. Switching to spectator doesn't stop the attraction, and since you can't pick the orbs up once they reach you they swirl around you like a swarm of angry bees.