To reproduce, attack a Zombie or Skeleton as fast as you can with any weapon(Basically spamming mouse1). The mob will not be able to do any damage to you. You can literally stand inside of it, and it will rarely do damage as long as it's constantly being hit.

This doesn't seem to happen with other mobs, only Zombies and Skeletons. This seemed to first appear during the SMP/SSP merge in 1.3.

EDIT: Was able to reproduce this in 1.4.4.

First picture is me constantly attacking the Zombie inside of a 2 block high hole, second one is when he was killed. I remained at the same health the entire time, which was 7 hearts