The previous mob AI prevented mobs from pathing across the bottom most plane of the map, the Void. The new mob AI does not take this into account and appears to treat the boundary between the bottom of the world and the Void as a solid plane which can be pathed across. It appears this is caused by the way which pathing is done. As there is nothing to detect below the point separating the world from the Void, the pathing checks to this point and assumes a solid surface exists, thus instructing mobs to attempt crossing and deleting themselves upon contact.

While normally this situation may not occur, maps which utilize the Void as a deterrent of the player must prevent mob access to areas low over the Void. Even at a height above the Void, mobs will conduct the normal calculations for dropping onto solid surfaces and attempt dropping onto the Void. Especially in this case, map makers attach large amounts of custom data to mobs, so the possibility of the mobs essentially deleting themselves is a frustrating thought.

Simplest way to test this: Create a superflat map and dig a 3x3 hole to the Void, leaving the center Bedrock block above it. Spawn a sheep or other mob on the Bedrock, then utilize anything which would normally cause the mob to move. It will walk off and be deleted upon contact with the Void, despite that there is no surface for it to walk on other than the non-collidable boundary..

Original Report: Before Jon's new AI, mobs would avoid wandering into the Void. The new AI does not take into account if the path they are taking is over open Void. It appears that currently Mob AI only takes into account the number of air blocks to fall, and since this can not be counted below the world limit, the AI essentially treats the Void as a solid floor. This means that if the Void is only a few blocks lower than the surface a mob is walking on, the mob will freely wander into the Void and die.

This can be tested by placing a block over open void and spawning a mob on it. The mob will normally simply walk off. This can also be tested by exploiting Villager's more predictable AI mechanics. First, create a classic Superflat world. Make a simple home underground with a door. Then place villagers inside. Break the bedrock directly in front of the door, and you will witness villagers freely wandering into the Void through the hole.

This should be easy enough to resolve - it has just gone widely unnoticed for quite some time.

Edit: Considering Jeb's declaration that all mobs will now use the new AI (https://twitter.com/jeb_/status/428840815549747200), I wish this issue reconsidered. Read comment below (Jan 30, 2014).