Only happens when villagers are asleep. Villagers end up on the floor and on stairs when sleeping. Sometimes they will fly across the map when asleep and appear back in their area when they wake up