So, in the multiplayer survival world my friend and I play on (hosted privately by me) we found a village that generated near a ravine, and decided to restructure it, moving the entire population (with the help of boats) into houses in the walls within the ravine. On the surface we erected a wall with a small gate around the entire ravine to protect our new sanctuary from Pillagers, and also to allow them to work some farms. The Villagers would claim their beds and job sites (all within the walls and ravine), work, gather at their meeting point, and return to their beds upon dusk. We needed to make everything safe and added some railings made of fences, because sometimes they would fall into the ravine and die. Eventually we solved that issue though, and the village was functional. So far so good.

However, after an hour or two of leaving the village alone, we noticed a lot of the villagers were missing, including a Master Armorer and Master Weaponsmith. Chaos ensured. Eventually we found them in the garden behind my friend's house that is outside the wall and on top of a hill, about 90+ blocks away from both their beds and their original job site blocks. I checked their data with '/data get entity UUID Brain'; some had claimed new job sites in my friend's house, some others had no job site but remembered their bed, and some had unclaimed their bed and their job site. We moved them back to their homes, made them claim new beds and job sites. Then it would happen again; I have noticed some also just getting stuck outside the walls at a certain point.

After doing some testing, I managed to narrow the problem down to the pathing of the Villager.

Example 1:

Even though the Librarian is on a pathable scaffolding and seems to know where his bed is, he is unable to path to it.

However, once I added stairs much closer to his job site and home, he was able to correctly path towards his bed.

Example 2:
The same would happen the other way around; upon trying to get to his lectern, the Villager would run towards and wait below the ledge on which his job site was.

After several failed attempts to reach his job site, he'd just become unemployed.

This confirms my hypothesis that the issue seems to be with the pathing range being too limited, also possibly the pathing itself being too simple, probably to ensure good performance and not cause lagging. This however seems to impact Villager behaviour significantly on more complex builds, as they can't find the path back to their job sites and beds after having wandered off too far.