Villagers wander away while being traded with

While trading with my farmers, they tend to wander off in the middle of selling to them, causing me to exit the GUI, then having to go find them to finish selling to them. Not sure if this happens with other types of villagers.