So far it seems like this only happens if you put your game on Peaceful. I went mining for coal to trade to my armorer (his profession was locked and there were 4 villagers) I came back up and there were two villagers. And this changed almost every time I relogged and changed difficulty intil it was stuck at two.