If wandering traders are under invisibility or it becomes day time, they don’t drink milk to clear the effect. They also don’t drink a potion if it becomes night time.