I put a mob into an inclosed space, and it starts to phase through the walls. The mob appears to travel through solid objects and into open areas. Then the mob gets "pulled" back into the enclosed space. For example, I put a squid into a two block tall, one block wide space full of water, and it starts phasing through the sides. I punched where the squid SHOULD have been, however, and the block underneath broke instead (I was in creative). Please note that you may have to be patient in duplicating this bug, but it still happens.
EDIT: This happens in ALL versions of Minecraft (I have seen it in the Xbox 360 version even)
ANOTHER EDIT: It happens to mobs other than squids, but only when you but too many of them in the same place (probably a different bug, but I'm just saying to test with squids)