I built an iron farm on my base and the golems spawn inside it as expected, they sometimes bug out and spawn in the wall which creates problems but that's not the issue here, there's other threads on that particular bug.
My issue is that 70 blocks away from my farm there's a hill, which for some reason is a favorite for the golems to spawn on. When I noticed it, there were around 30 golems on that hill just hanging around. After removing the golems and some of the trees there, I also removed some area around and under the farm itself, but still, every now and then, I see golems popping up. There is no villager there, there are no buildings, no structures, no blocks placed by me at all, it's just a hill that I haven't done anything with.
Distance from the farm, around 70 blocks.
Farm has around 80 villagers, workstations and beds.
Images attached show the farm and the distance the golems spawn from it.