I killed some villagers that were on the outskirts of the village and were too much of a hassle to move back, i killed quite a few, but I thought the villager happiness had a minimum happiness level. I've traded quite a bit but I can't get the golems to stop attacking me. I've even tried letting them kill me. No luck.