I punched a villager in Hard mode and the golem just stared I tried again and it just kept standing on the edge of a trapdoor, I then broke it and it finally attacked. I punched another villager to see what would happen,then when it rushed towards me i set the difficulty to peaceful and it kept attacking and throwing me around.