Tested with some accuracy: any time a chunk, at least in SMP, updates without a player in range, baby animals despawn or disappear.
Tested officially once, unofficially (sort of by mistake) twice and encountered several times. Confirmed for pigs and chickens.
Source of baby animal irrelevant. Bred and Egged chickens, for instance, both disappear.