Villagers put into a Villager breeder are vanishing or despawning

What I expected to happen was...:
I followed a guide on how to build a villager breeder from ImpulseSV and I expect the villagers to breed together and stay in the breeding module without despawning or vanishing

What actually happened was...:
Villagers will breed once or twice and then one or both of the villagers will vanish therefore breaking the breeder. This has happened a couple times now.

Edit: Name tagged Villagers also vanishing but breeder is still running so starting to think this is a render bug 

Steps to Reproduce:
1. Build this breeder from ImpulseSV https://www.youtube.com/watch?v=VTseLpa9Iyc&t=648s
2. Travel away from the breeder until it is outside render distance
3. Sleep for a couple nights then return to the breeder

I am unsure if these exact steps will recreate the issue but these are what I was doing that caused the villagers to vanish