If you switch to a different hotbar item or lose track of an animal following you while holding their associated food, they will not start following again until said item is dropped on the ground and picked up again.