I haven't hit or killed any villagers or golems and I cure a lot of zombie villagers. No matter what I do the prices always rise bit by bit over the course of an hour. Whether I leave or come back or trade continuously doesn't matter, and even cured zombie villagers prices will increase to default in around an hour. 

Even a cured zombie villager that I hadn't even given a job site block until later had all default prices when I finally started trading.

I asked on reddit and I think someone mentioned supply/demand but I think the wiki says that's only bedrock and even if it is Java, I haven't traded with some of them in days/weeks (in mc)