What I expected to happen was...:
Villagers would be able to descend with a ladder when the point they want to reach is accessible only through a ladder.

What actually happened was...:
They cannot go down, only up.

Steps to Reproduce:
1. Find any generated village.
2. Find a cubic house (the one with a ladder to the top with fences on the roof)
3. Either wait for a villager to walk up or make one.
4. He'll stay there forever (even if there's a zombie up there).