Some villagers despawn in trading halls and in vegetable farms where they are protected and illuminated. I cannot find a common theme between those that do or do not despawn however it appears to have occurred after my breeder had generated about 30 additional villagers.  (I had disabled the breeder thereafter)