Villagers have despawned from roaming away from them. Possibly from entering the Nether or the End. It has happened while staying in the over world. They're kept safely in a house with beds,a bell, job blocks and most are at max trade level. 

Another world I have also has the same issue. These villagers roam free inside a walled-in well lit area guarded by iron golems villagers have spawned. It's possible these may have died but highly unlikely. I've never seen any mobs spawn while building.