Every time I find a village and spend a couple of days there the population decreases dramatically. After one day there might be half as many villagers as there was when I first got there and then after two days, only like 3 and after 3 days they're all gone. The iron gollum doesn't always disappear and 4 out of 5 times I've put 3-5 more iron gollums down. 

I've currently found another one and have put a fence round the whole village but after just one day of being there (in game days I'm talking about) there is only 3 villagers left. They're not even dying as the past couple of times I haven't even been out of the village or more than 30 blocks out of the village.