Villagers some how lose their jobs, and it's currently unknown of why they do.
However this house layout may help give context.
Certain Villagers such as Fletcher, and Tools Smith become unemployed, and act like they have no home and wander off And even when they're a few blocks away, it doesn't have much changes.