I made a village breeder with a trading hall next to it. When it turned night, they all went to their beds as usual, but then they just started moving away. Like they were still laying down, but started moving horizontally away from their bed. I would destroy the bed and they would sometimes return, but not every time. Villagers moving towards their bed would move other villagers already lying down as well.