leveling villagers does not always work and resets the same level so your about to get to lv 3 and all the sudden your 2 again with no experience.  has happened at all levels for me sometimes have to level a villager 2 times for every level up to Max 5

happens alot but possibly not every time im not sure seems 100%

it also affected 1.16 and 1.16.1