I'm not sure why is this happening,but I have been having an issue where a villager doesn't recognize their job site. I had this happen to a Tool Smith and a Librarian. Whenever I break their job site and place it back down then put the librarian next to it another villager takes the job and the librarian acts if they are unemployed and starts wondering around.