Build my home base, then went about 100+ blocks away in two different directions to build custom villages in survival. Cured zombies and populated villages successfully, but instead of picking beds or workstations that were in the village, they sometimes run the distance and invade my house instead.

For example, I did not place any barrels in my first two villages, but but I have had several fishermen come to my house. They seem to have problems opening the door, possibly because some of the barrels I have are in basements and they don't know how to pathfind to those locations, so they just stand around outside roughly over the location of the underground barrels and eventually get attacked by zombies.

Also had similar problem with brewing stands and clerics. I have not placed brewing stands in either village, but baby villagers are growing up as clerics and running to my house. Sticking villagers into boats and bringing them back to their village does not fix the problem. They still return.

Thought at first problem was that babies were roaming very far from home and then growing up closer to my house than to the village, so I completely enclosed the village. However, I have fishermen and clerics showing up in the village still. One fisherman just stands at the fence corner closest to my house and stares into the distance.

EDIT TO ADD: Was going to include a pic of this fisherman, couldn't find him, assumed he'd died in a recent raid. But later he emerged from my mine. Not sure how he escaped the fenced area, but it's definitely him, because he's earned some experience from trading.

Tried also to place a barrel in the village to give the fisherman a workstation they could reach. Did not fix the problem.

As for beds, I kept my bed in an upstairs loft. A fisherman who actually made it into my house took my bed, but could not leave the upstairs after waking up the next day. They stayed there for quite some time before a zombie somehow invaded.

Tried breaking all beds, or placing beds in basement where they could not be reached. Still see the "sparkly stars" over the bed indicating a villager is trying to reach it to sleep.