I noticed that villagers tend to pathfind to the block above their job site, and if there is a block above their job site, they will continue targeting upwards, and they often end up on the roof of the building. (See pictures for examples)

If the block above is broken, they will continue to pathfind to the job site.