I have an underground base, that I started breeding villagers within. I have enough beds and workstations. The mate regularly and free roam around my base. As do the iron golems, and cats. At almost all times there is one of the three mobs stuck at a dark oak fence gate. I belive the path finding is making them think going through  the gate and through a wall is the shortest route. However I have a nice 3x3 hallway right around the corner.