Even though their is no job site near them, and have not been traded with, they will not become jobless. This didn't seem to affect all of them. I set the villagers profession then broke the job site blocks during the night. I then gave them their job site blocks back after they became jobless, then broke them again. There are no job sites anywhere near them.