I've had to clear out underground caves filled with trader llamas due to this. Note that I never interacted with these llamas (such as riding, leashing, nametagging, feeding), so it shouldn't have permanence. Of course, trader llamas that have been interacted with shouldn't despawn, but please despawn if otherwise.