Steps to reproduce:

1. Create a village OR find a generated one (I've found that it occurs in villages with more than 5 villagers tho)
2. Ring the village bell (this should force them to pathfind to their "house" and cause a massive lag spike but sometimes it doesn't.
3. Place workstations far from their beds. I find it when they wake up to work they cause lag as well as them pathfinding to their beds.
4. Have a single zombie chase them in a closed environment along with their bed and workstation.

Additional notes:

I think it's related to their pathfinding calculations, since if I lock them down in a single block, and even with other mobs swarming within my spawning/despawning sphere it doesn't lag (I get like 5-70 in ServerTime when they're not under any of the situations above, and 300-1800 when I test it)

Here's a youtube link for case 4 but it still lags even if the villagers are not moving at all: https://youtu.be/M2s6v5LEKrM