In some large villages most golems mill about in the same small area all day. I originally supposed that he presence of a cave system with mobs underground might be attracting the golems, but the golems still mill around in the same place even when peaceful mode is activated and there are no mobs. This is noticeable in villages where villagers and golems have multiplied quickly and there are a great many of them. In a village with only one or two golems I wouldn't even notice if the golems were always staying in the same area.