Villagers won't reduce trade prices

No matter how much I trade with villagers in a village, they still hold resentments for damaging them or their golems. They're supposed to gradually forgive you once you trade with them repeatedly, but even new villagers charge extra. The main issue is that there's no way to reverse this, and their prices are permanently elevated.