After trading with the villagers they can unlink with their work blocks. I've done some testing and it really only becomes a big problem if you are trying to make a villager hall. 1.18 + seems to be the only affected version, I've done testing in both realms and private worlds and I have gotten the same result.