So I've been seeing villagers try to go somewhere and then they stop and turn around and end up being in a loop of going back and forth. This is most clearly seen at night time when the villagers try to find a bed (as seen in the video). Also, villagers will sometimes all try to go for one bed which ends up making there be like 5 villagers surrounding a bed.

EDIT: I've done the test without doors and realized it is 100% due to the doors. When I removed the doors the villagers went to the beds perfectly with no trouble. So, the real problem is villagers going to open doors to get to stuff.