Mobs can still track you and hurt you even if you have the invisibilty effect active. They will track you regardless of if you hit it before or not, or if it was previously tracking you.

Test:

1. Summon Zombie.

2. Use invisibility splash potion.

3. Wait a while or punch zombie and it will follow and damage you. Notice they can track you even if you do not hit it before.

Apparently this was fixed according to MC-18284 but it still is occuring...