The bug

Mobs look at invisible players or other mobs even if they are not on the same team or the team option seeFriendlyInvisibles is disabled.

How to reproduce

1. Summon an invisible villager

   /summon villager ~ ~ ~ {ActiveEffects:[{Id:14b,Amplifier:0b,Duration:1200,ShowParticles:0b}]}
   

2. Summon a second villager
3. Switch to Spectator mode to make sure they are not looking at you

Code analysis

Based on 1.11.2 decompiled using MCP 9.35 rc1

The methods net.minecraft.entity.ai.EntityAIWatchClosest.shouldExecute() and net.minecraft.entity.ai.EntityAIWatchClosest.continueExecuting() ignore whether or not the entity is invisible.